:W&Bá¾þì>¬M zy9gyq0rª; 9 /0.28.0+a4305a7c469ab281e827fb7f0d5485a00923f9e60.65.0 ð(9}‚" 6xjxphczdgxp zy9gyq0rŠÙ zy9gyq0rescheuller-uc-san-diego spatop-sweep" _wandb‚{}  batch_size‚2048  dropout‚0.149327849253641  hidden_dim‚64 ( learning_rate‚0.00023666733111463696  num_embedding_layers‚10Bdpfx_blocks_zy9gyq0rb78avrrzyj!dp-spatop-sweepagent-blocks-fzp76Š éŸåÒ¾÷ßš3 (H¨¸hp"3.12.10*0.28.0j linux-x86_64'¨/©; zy9gyq0r*  _wandb‚ {"runtime":0}  _runtime‚0Š€ƒ]Ò× -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 éŸåÒÀóÙ‘*-of*>/data/spatop/options_files/sweep_fixed_dp/config_zy9gyq0r.json*-l* /scratch/logs*-n*dpfx_blocks_zy9gyq0r2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb!dp-spatop-sweepagent-blocks-fzp76r/opt/conda/bin/pythonº 4bbryvwyl2bh63ecziocc4k7dcxlsl3næPò2  requirements.txt~?OòW zy9gyq0rZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.0j linux-x86_64"áY zy9gyq0rZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.0j linux-x86_64Ê×Û* zy9gyq0rb trainer/global_step2æDò\  zy9gyq0rZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.0j linux-x86_64 žÜé/  zy9gyq0rb*"trainer/global_step20ÿí×J  zy9gyq0rj9 êŸåÒÈòÙP*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ¬|íƒÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€€›¢Ê#ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º 4bbryvwyl2bh63ecziocc4k7dcxlsl3nä.î’  zy9gyq0rj€  êŸåҀǿnî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ 3$¸X[ zy9gyq0rjÉ êŸåÒ¸††o·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 ô:‹ p zy9gyq0rj_ êŸåÒÈ•›oPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). .¶µNz zy9gyq0rji êŸåÒȃ¡oZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ð¸zp zy9gyq0rj_ êŸåÒØȤoPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ´W“šs zy9gyq0rjb êŸåÒˆ³¦oSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). žò¨m zy9gyq0rj\ êŸåÒ£âoMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). 4ª + zy9gyq0r*™  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output",6ãê zy9gyq0rjØ êŸåÒ ¯òqÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. êa?®M zy9gyq0rj» êŸåÒ˜íúq«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. @vIÓë zy9gyq0rjÙ ìŸåÒøõ×”È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ótP zy9gyq0rj¾ ìŸåÒð¤å”­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ÍÑÇ&š‚(: ùŸåÒ“ßÚA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚40.0 gpu.0.powerWatts‚107.503' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.83433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|WË3<‚(:² ùŸåÒ‘öÕB proc.cpu.threads‚16 network.recv‚39693 disk.md0.out‚ 0.42578125 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 6.30078125 disk.dm-1.out‚ 0.0859375 disk.nvme1n1.in‚ 0.0390625 network.sent‚32262 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme1c1n1.in‚ 0.0390625" disk.nvme1c1n1.out‚ 0.888671875$ memory_percent‚87.31179237365723) proc.memory.percent‚17.14920997619629 cpu‚25.285391965067568 disk.nvme0c0n1.in‚ 0.0390625" disk.nvme0c0n1.out‚ 6.408203125) disk./.usagePercent‚74.16166657443311 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 2078.8359375" proc.memory.rssMB‚ 2809.7265625% disk./.usageGB‚1138.5669860839844 disk.nvme0n1.in‚ 0.0390625 disk.nvme0n1.out‚ 6.408203125 disk.nvme1n1.out‚ 0.888671875A"9¯P‚(:Æ ˆ åÒ¢±þA disk.md0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2809.74609375 proc.cpu.threads‚16& disk.nvme0c0n1.out‚111.21435546875 disk.nvme1n1.in‚ 0.0390625" disk.nvme1n1.out‚ 8.02099609375 disk.nvme0c0n1.in‚ 0.0390625$ disk.nvme1c1n1.out‚ 8.02099609375 disk.dm-0.in‚0.078125 disk.dm-1.out‚ 0.31689453125$ memory_percent‚87.31565475463867' proc.memory.availableMB‚ 2078.203125) proc.memory.percent‚17.14932918548584 network.sent‚42573) disk./.usagePercent‚74.16087245149151% disk./.usageGB‚1138.5547943115234 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.0390625 disk.md0.out‚ 0.75390625! disk.dm-0.out‚117.30712890625 cpu‚25.39586183801046 network.recv‚43688 disk.rbd0.out‚0$ disk.nvme0n1.out‚111.21435546875 disk.nvme1c1n1.in‚ 0.03906252Ë“Q›‚(:‘ ˆ åÒÐÛ°B gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚41.0 gpu.0.powerWatts‚108.406' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.135333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+vjÃ3b/ lr-AdamW/pg1"trainer/global_step2:JP#©Îù3b/ lr-AdamW/pg2"trainer/global_step2:JP°¦Ò-) _wandbruntime‚32  _runtime‚32ú “+ ½ _step‚0 ' lr-AdamW/pg1‚2.8991748061543026e-05 ' lr-AdamW/pg2‚2.8991748061543026e-05 trainer/global_step‚49 % _timestamp‚1.7842380911158977e+09 _runtime‚ 32.979923698RžP­Ä!¿ % _timestamp‚1.7842380911158977e+09 _runtime‚ 32.979923698 _step‚0 ' lr-AdamW/pg1‚2.8991748061543026e-05 ' lr-AdamW/pg2‚2.8991748061543026e-05 trainer/global_step‚49.óöêk"bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPà â·g#bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPÉnˆm$bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPìša%b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÛ~Ì k&bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÖS‚b'b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPB˜Á¤r(bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPjùüi)be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP `Ðf*bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP½9c+b_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPfFúÉg3bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP«2°i4be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPYÔ––p5bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPIäÈ·s6bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPIk“^7bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPNH p8bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP’Ni9be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP'·y‘f:bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP*Þ»%f;bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP"ó<(nbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPZðp]g?bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP-Öêg@bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP´*ÑÜaAb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP<]ªHjBbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP틤lCbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP•z\=nDbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP!|¼!qEbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPŠìî¶lFbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPDýn«mGbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPìùFfHbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP@ÙÒhIbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPtâý_sJbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÅðÚÕsKbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPuØŽcoLbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPF2»KjMbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP•Gª…tNbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP_U nObj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPR/H–jPbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP›Uo}oQbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPqõ.M^RbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP¯ >lSbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPzáqTbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPéд–gUbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPB hÙnVbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPµ™ZîkWbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÈHqXbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPçš9¸dYb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPõ¤'bZb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPUñ®£i[be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP†Êe>e\ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÀƒÈp]bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP_Q!Œk^bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP\ô?e_ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP}ðžÂl`bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPOyk÷nabj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPëdƒobbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP3ûéèncbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPWÁ6vkdbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPèä³]ebY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP×þ`fb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPêÕ¥!sgbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPp\ž`hb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP«tÜTgibc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPܱMæjjbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPhÂikbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP¾ÃÇvlbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPlP_smbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPñöãixnbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPFI‚WobS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPìÌëôqpbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP™pç}rqbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPz´„jrbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP¹\Z–hsbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP´ßhtbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPû¬á€rubn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPÎonpvbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPÅܲcwb_ !¡jbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP’]pµe€b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPë¸Brbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPìBeÚh‚bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP¶OËÕqƒbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPÒÊõ›n„bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPps–g…bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP¨ †`†b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPJ„#g‡bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP3¾Žøpˆbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPM•jq‰bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP5^/qoŠbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP u¾®o‹bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP÷ÕPpjŒbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP±Ãfba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPâCË7pŽbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPÖð áobj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP§lŠtbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP@ñwm‘bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPÈ+}Fu’bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPU o“bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP7¾Þo”bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPΈ¹k•bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPM0r¡`–b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPì«nh—bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP¬ÏÂxm˜bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP²ÈAf™ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP5Bwšbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPCÀTHn›bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¾7ªilœbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPü¡?u.) _wandbruntime‚32  _runtime‚32o zršžýN QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3816769769984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 trainer/global_step‚49 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117040640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 $ _timestamp‚1.784238091116925e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269120000 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚369245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12021 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301696512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 132997632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11950 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚385423 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚801 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882639 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3772951413700 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415743 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2893712427008 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚490284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3732963929872 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 39987483828 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3776652512768 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚490284 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 41567936000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40011737600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3776652512768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269120000 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚369431 _runtime‚ 32.984374282 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2893923577344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 40100217856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3816664250368 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40011737600 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚500076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211150336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3776669552128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 41684976640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385423 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3816664250368 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚254101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3732946890512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 40100217856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2852144491008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚254097†a‡'ŸO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 40100217856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3816664250368 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40011737600 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚500076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211150336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3776669552128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 41684976640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385423 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3816664250368 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚254101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3732946890512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 40100217856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2852144491008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚254097 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3816769769984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 trainer/global_step‚49 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117040640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 $ _timestamp‚1.784238091116925e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269120000 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚369245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12021 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301696512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 132997632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11950 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚385423 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚801 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882639 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3772951413700 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415743 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2893712427008 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚490284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3732963929872 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 39987483828 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3776652512768 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚490284 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 41567936000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40011737600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3776652512768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269120000 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚369431 _runtime‚ 32.984374282 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2893923577344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682'­‰ng bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPª6h|`¡b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPtëàŒj¢be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP¡¦se£b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP1aÏá_¤bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPØಣn¥bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÞö f¦ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP/,3ˆd§b_ X|Ôq»bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPýýѦh¼bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPX Ðg½bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP‡ºci¾bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPcÅÿp¿bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPõŸKPiÀbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP,1ÁuÁbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP"ÚÖmÂbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP)@}ùhÃbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP2 FmÄbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPÈfç2lÅbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÍK°¶sÆbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPù¢ÝaeÇb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPä/YhÈbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPÇJ"7iÉbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPBÉg—uÊbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPسÑ\ËbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP3ʃmÌbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP㎞nÍbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP´ kƒqÎbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPó’¬mÏbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPI©3ó^ÐbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPê ¯^eÑb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP1[*kÒbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP¹TÕåkÓbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPžŠ#fÔba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP¢_¢ rÕbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP…²ÂjÖbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP'×Iÿ^×bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPÞƒÃwjØbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPa3AeÙb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPA>hÚbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPlN^¼mÛbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPx`¯]ÜbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP¦#7ÔfÝba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP’yÙÔ]ÞbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPAzçøaßb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPµy@«màbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP¿¸Ágább ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPþ0§kâbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP!}+lgãbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPÌ”žWeäb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPüÎAçmåbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP3xsæbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP†âiçbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPòcFVèbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP± VDféba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPépêbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP<¤Œmëbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPéÐF­bìb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP±~“mlíbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPâ‰drîbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPeŠúqkïbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP.›±°fðba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP+± ßqñbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPNrZjòbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP¶kR¦móbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPÈ/{fôba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP'µðiõbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPi'°Pdöb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPo¬ûjbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPm NÖh‚bc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPŽ ›øhƒbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPPÿñ˜l„bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP ³·c…b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPÌI(i†bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPy×;k‡bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP’I!pˆbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP—™—ƒi‰bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP)³qŠbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP9D·Jn‹bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPáfÌDrŒbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPõª`J_bZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP¯ \™dŽb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP g%ôh”bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP'dbb•b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP‚#ûsm–bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPÂ9Fn—bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPtX/r˜bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP'òîÎl™bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPai_þlšbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP’„ì.›) _wandbruntime‚33  _runtime‚33uà°2œˆM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 % _timestamp‚1.7842380915109184e+09 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚509910 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2950869958656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 40797453312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3807254455264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209046016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3807237415904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.curÐi­\ rent‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3892599632384 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3848025433804 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3847921609434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚ 33.375058171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 42378693632 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393052 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚259153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 40770978540 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3892705159168 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚117568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚509910 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚376542 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130CT‰ù’&ŒM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209046016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3807237415904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3892599632384 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3848025433804 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3847921609434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚ 33.375058171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 42378693632 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393052 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚259153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 40770978540 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3892705159168 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚117568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚509910 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚376542 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 % _timestamp‚1.7842380915109184e+09 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚509910 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2950869958656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 40797453312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3807254455264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N loss/SRqqt2/assignment_loss"trainer/global_step2:JP?â=@¡b; loss/FRt1/detection_loss"trainer/global_step2:JP/*P†@¢b; loss/FRt2/detection_loss"trainer/global_step2:JPhxoÿ-£b( epoch"trainer/global_step2:JP¾Ñ%ãC¤b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP;«µB¥b= loss/SRqqt1/detection_loss"trainer/global_step2:JP½}¥B¦b= loss/SRqqt2/detection_loss"trainer/global_step2:JPM›2@§b; loss/FBt1/detection_loss"trainer/global_step2:JP]­×óA¨b< loss/FRt1/assignment_loss"trainer/global_step2:JPš¬À+A©b< loss/FRt2/assignment_loss"trainer/global_step2:JP£/¶Aªb< loss/FBt1/assignment_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JP³¢Ñz.¬) _wandbruntime‚33  _runtime‚33 ñ~&ô­î /loss/SRqqt2/assignment_loss‚4.2101176933006 .loss/FRt1/detection_loss‚1.053372962759127 /loss/FRt2/detection_loss‚1.6868974080127246 epoch‚0 1loss/SRqqt1/assignment_loss‚4.235351108524897 0loss/SRqqt1/detection_loss‚3.082954252436417 0loss/SRqqt2/detection_loss‚2.162741152855454 .loss/FBt1/detection_loss‚19.85883549983437 % _timestamp‚1.7842380915119603e+09 _runtime‚ 33.375931882 /loss/FRt1/assignment_loss‚4.532024197132662 /loss/FRt2/assignment_loss‚5.323589125151934 /loss/FBt1/assignment_loss‚2.233532559227299 %loss/total_loss‚63.51397559308925 trainer/global_step‚49 _step‚3 0loss/FBt2/assignment_loss‚2.1771820718591863 /loss/FBt2/detection_loss‚12.957377561994575un=“ø®ò .loss/FBt1/detection_loss‚19.85883549983437 % _timestamp‚1.7842380915119603e+09 _runtime‚ 33.375931882 /loss/FRt1/assignment_loss‚4.532024197132662 /loss/FRt2/assignment_loss‚5.323589125151934 /loss/FBt1/assignment_loss‚2.233532559227299 %loss/total_loss‚63.51397559308925 trainer/global_step‚49 _step‚3 0loss/FBt2/assignment_loss‚2.1771820718591863 /loss/FBt2/detection_loss‚12.957377561994575 /loss/SRqqt2/assignment_loss‚4.2101176933006 .loss/FRt1/detection_loss‚1.053372962759127 /loss/FRt2/detection_loss‚1.6868974080127246 epoch‚0 1loss/SRqqt1/assignment_loss‚4.235351108524897 0loss/SRqqt1/detection_loss‚3.082954252436417 0loss/SRqqt2/detection_loss‚2.162741152855454>¦1›¯‚(: — åÒí»ÓA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚43.0 gpu.0.powerWatts‚109.099' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.36633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àßP°‚(:Å — åÒÖâA network.recv‚50752 disk.nvme1n1.in‚ 0.0390625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.0390625$ disk.nvme1c1n1.out‚ 8.80908203125 network.sent‚54270) disk./.usagePercent‚74.16088338434803" disk.nvme1n1.out‚ 8.80908203125 disk.md0.out‚1.109375 disk.nvme0c0n1.in‚ 0.0390625# memory_percent‚87.3145341873169* proc.memory.percent‚17.149686813354492 proc.cpu.threads‚16 disk.dm-0.in‚0.078125! disk.dm-0.out‚123.47900390625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚117.48681640625$ disk.nvme0n1.out‚117.48681640625 disk.nvme1c1n1.in‚ 0.0390625% disk./.usageGB‚1138.5549621582031 disk.dm-1.out‚ 0.43408203125) proc.memory.availableMB‚ 2078.38671875" proc.memory.rssMB‚ 2809.8046875 cpu‚25.44450380008628qhɱ‚(:’ ¦ åÒà¤ùA gpu.0.gpu‚59.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚45.0 gpu.0.powerWatts‚199.312' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.43733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@(iÎH²‚(:½ ¦ åÒ…æ‰B disk.md0.in‚0 disk.dm-0.in‚0.078125* proc.memory.percent‚17.149782180786133 disk.md0.out‚ 1.3515625 disk.dm-0.out‚135.3837890625 disk.dm-1.in‚0# memory_percent‚87.3173475265503 cpu‚25.483055367329715 disk.nvme1n1.in‚ 0.0390625 network.sent‚101057 network.recv‚54767) disk./.usagePercent‚74.16092214629386 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 128.12890625 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme0n1.in‚ 0.0390625" disk.nvme1n1.out‚ 10.8701171875 disk.dm-1.out‚ 0.7109375) proc.memory.availableMB‚ 2077.92578125" proc.memory.rssMB‚ 2809.8203125 proc.cpu.threads‚16 disk.nvme1c1n1.in‚ 0.0390625$ disk.nvme1c1n1.out‚ 10.8701171875% disk./.usageGB‚1138.5555572509766 disk.rbd0.in‚0! disk.nvme0n1.out‚ 128.12890625cL¿.³) _wandbruntime‚64  _runtime‚64NtÙS´¼ _runtime‚ 64.081795685 _step‚4 ' lr-AdamW/pg1‚5.8575164450872645e-05 ' lr-AdamW/pg2‚5.8575164450872645e-05 trainer/global_step‚99 $ _timestamp‚1.784238122217838e+09Z¶Ä=ƵÀ ' lr-AdamW/pg1‚5.8575164450872645e-05 ' lr-AdamW/pg2‚5.8575164450872645e-05 trainer/global_step‚99 $ _timestamp‚1.784238122217838e+09 _runtime‚ 64.081795685 _step‚4»àÿ.¶) _wandbruntime‚64  _runtime‚64ºÐ~Ɉ'·‚O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚983136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7613497505792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚506897 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7613497505792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚225979 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 82209615872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7534152867328 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚766871 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 79226014636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚172 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚766873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7534152867328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7447490199472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚506901 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5741445365760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110749184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7526716214108 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842381222187262e+09 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚732876 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 79450158080 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7526612389850 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7534135827968 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 79361677824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7534135827968 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚992928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7613603025408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 79361677824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7447473160112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚766871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 82320365056 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚992928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7613497505792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚506897 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7613497505792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚225979 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 82209615872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7534152867328 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚766871 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 79226014636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all)q}© .current‚172 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚766873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7534152867328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7447490199472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚506901 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5741445365760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110749184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7526716214108 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842381222187262e+09 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚732876 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 79450158080 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7526612389850 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7534135827968 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 79361677824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7534135827968 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚992928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7613603025408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 79361677824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7447473160112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚766871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 82320365056zd÷?.¹) _wandbruntime‚64  _runtime‚64gHÛ”&ºŽM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1002762 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5798186698752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚511957 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 79922724338 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚740150 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 83018079744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7521780724864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 80235880960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚992956 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774502 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7609285494272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7601790234212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7521763685504 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5715168619008 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 80009509348 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774500 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 80147393536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7609302533632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5798399939072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1002762 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚740329 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚511953 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7609285494272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774502 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 80235880960 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚228197 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7689538414592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 % _timestamp‚1.7842381226121187e+09ýçc,˜&»’M F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5715168619008 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 80009509348 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774500 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 80147393536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7609302533632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5798399939072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1002762 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚740329 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚511953 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7609285494272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774502 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 80235880960 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚228197 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7689538414592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 % _timestamp‚1.7842381226121187e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1002762 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5798186698752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚511957 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 79922724338 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚740150 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 83018079744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7521780724864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 80235880960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚992956 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774502 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7609285494272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7601790234212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7521763685504Åçª.¼) _wandbruntime‚64  _runtime‚64š2å­÷½ñ 1loss/SRqqt1/assignment_loss‚4.247023904195395 1loss/SRqqt2/detection_loss‚1.7706878965700568 .loss/FBt1/detection_loss‚39.78897109627724 epoch‚0 trainer/global_step‚99 _step‚7 /loss/FRt1/assignment_loss‚4.494048904296786 1loss/SRqqt2/assignment_loss‚4.165491965791898 0loss/FBt1/assignment_loss‚2.0297852920161352 $loss/total_loss‚78.4843314472798 _runtime‚ 64.476731205 /loss/FRt2/assignment_loss‚5.331589609500801 /loss/FBt2/assignment_loss‚1.892326671760399 1loss/SRqqt1/detection_loss‚3.5656482032850705 -loss/FBt2/detection_loss‚9.30767825264093 0loss/FRt1/detection_loss‚0.43305277175051765 /loss/FRt2/detection_loss‚1.4580268791945765 % _timestamp‚1.7842381226130967e+09HF½×û¾õ 1loss/SRqqt1/assignment_loss‚4.247023904195395 1loss/SRqqt2/detection_loss‚1.7706878965700568 .loss/FBt1/detection_loss‚39.78897109627724 epoch‚0 trainer/global_step‚99 _step‚7 /loss/FRt1/assignment_loss‚4.494048904296786 1loss/SRqqt2/assignment_loss‚4.165491965791898 0loss/FBt1/assignment_loss‚2.0297852920161352 $loss/total_loss‚78.4843314472798 _runtime‚ 64.476731205 /loss/FRt2/assignment_loss‚5.331589609500801 /loss/FBt2/assignment_loss‚1.892326671760399 1loss/SRqqt1/detection_loss‚3.5656482032850705 -loss/FBt2/detection_loss‚9.30767825264093 0loss/FRt1/detection_loss‚0.43305277175051765 /loss/FRt2/detection_loss‚1.4580268791945765 % _timestamp‚1.7842381226130967e+09é+¶R›¿‚(: µ åÒïØÌA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚46.0 gpu.0.powerWatts‚214.547' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.51566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0apû¬AÀ‚(:¶ µ åÒ¼¢ëA$ disk./.usageGB‚1138.555435180664 disk.dm-1.in‚0 disk.dm-1.out‚ 0.77734375 disk.nvme0n1.in‚ 0.0390625$ disk.nvme1c1n1.out‚ 11.6123046875 disk.nvme1n1.in‚ 0.0390625" disk.nvme1n1.out‚ 11.6123046875 disk.dm-0.in‚0.078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚143.6884765625) proc.memory.availableMB‚ 2077.69140625* proc.memory.percent‚17.149829864501953 proc.cpu.threads‚16 network.sent‚105309( disk./.usagePercent‚74.1609141951255 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625 disk.md0.in‚0 disk.md0.out‚1.703125# memory_percent‚87.3187780380249! proc.memory.rssMB‚ 2809.828125 network.recv‚58813! disk.nvme0c0n1.out‚ 136.515625 disk.nvme0n1.out‚ 136.515625 cpu‚25.508802518912066dÍ›—œÁ‚(:‘ Ä åÒÞÀïA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.794' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.931333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðé;óF‚(:» Ä åÒÝÄþA$ memory_percent‚87.31834888458252* proc.memory.percent‚17.149829864501953 cpu‚25.529277054689842 network.sent‚153336 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 146.41015625 disk.dm-0.in‚0.078125 proc.cpu.threads‚16) disk./.usagePercent‚74.16091866765771 disk.dm-1.in‚0" disk.nvme1c1n1.out‚ 13.12109375 disk.nvme1n1.in‚ 0.0390625! proc.memory.rssMB‚ 2809.828125% disk./.usageGB‚1138.5555038452148 disk.dm-1.out‚ 0.990234375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.0390625! disk.nvme0n1.out‚ 146.41015625 disk.md0.out‚ 2.32421875 disk.dm-0.out‚ 153.568359375) proc.memory.availableMB‚ 2077.76171875 network.recv‚63193 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625 disk.nvme1n1.out‚ 13.12109375 disk.md0.in‚0˜yN.Ã) _wandbruntime‚95  _runtime‚95 )×Âļ trainer/global_step‚149 % _timestamp‚1.7842381537176106e+09 _runtime‚ 95.581556349 _step‚8 & lr-AdamW/pg1‚8.815858084020226e-05 & lr-AdamW/pg2‚8.815858084020226e-05K†ž±ÆÅÀ _runtime‚ 95.581556349 _step‚8 & lr-AdamW/pg1‚8.815858084020226e-05 & lr-AdamW/pg2‚8.815858084020226e-05 trainer/global_step‚149 % _timestamp‚1.7842381537176106e+09'„$.Æ) _wandbruntime‚95  _runtime‚95U(ÄbgÇ¡O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 118711618048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1475988 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚336180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚ 95.585346536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11291619143168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11410330761216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_‰êî@train_batch_start/allocation.large_pool.freed‚1148321 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 118800098304 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1475988 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11291619143168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚149 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148321 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 122782950912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚336359 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11162016469072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1096060 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 118711618048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 _step‚9 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1485780 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 118800098304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8466011400704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 % _timestamp‚1.7842381537184546e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8588923975168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223733248 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚759697 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11291636182528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8588700241920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚759701 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11161999429712*|[Ï«'È¥O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1475988 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚336180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚ 95.585346536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11291619143168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11410330761216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148321 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 118800098304 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1475988 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11291619143168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚149 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148321 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 122782950912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚336359 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11162016469072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1096060 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 118711618048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 _step‚9 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1485780 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 118800098304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8466011400704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 % _timestamp‚1.7842381537184546e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8588923975168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223733248 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚759697 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11291636182528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8588700241920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚759701 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11161999429712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 118711618048 æÂÎ.É) _wandbruntime‚95  _runtime‚95:=®;²&ʬM J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1103095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1103267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 119248040156 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 119497333760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 95.978907659 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8645645278720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1485808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11355451210250 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11355555034620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1155950 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1155952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8522149128704 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 trainer/global_step‚149 _step‚10 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11366785848832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 119585821184 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1155952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8522149128704 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 trainer/global_stepÿÎo¤‚149 _step‚10 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11366785848832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 119585821184 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1103095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1103267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 119248040156 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 119497333760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 95.978907659 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8645645278720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1485808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11355451210250 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11355555034620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 YƒŸŽ.Ì) _wandbruntime‚95  _runtime‚95¿2$üÍö /loss/FRt2/detection_loss‚1.4101772889658117 1loss/SRqqt2/detection_loss‚1.4567607650413747 %loss/total_loss‚85.72625057775451 _runtime‚ 95.97935884 /loss/FRt2/assignment_loss‚5.363601278398455 1loss/SRqqt2/assignment_loss‚3.828344828051949 0loss/SRqqt1/detection_loss‚3.581411051057594 .loss/FBt1/detection_loss‚51.91526733062886 epoch‚0 trainer/global_step‚149 _step‚11 0loss/FRt1/assignment_loss‚4.5797628160933375 .loss/FBt2/detection_loss‚7.884288258142944 % _timestamp‚1.7842381541157153e+09 2loss/SRqqt1/assignment_loss‚3.8956438702596743 0loss/FBt1/assignment_loss‚0.8923708421212656 0loss/FBt2/assignment_loss‚0.4664016754305276 0loss/FRt1/detection_loss‚0.45222057356272866©Fõ3Îú /loss/FRt2/detection_loss‚1.4101772889658117 1loss/SRqqt2/detection_loss‚1.4567607650413747 %loss/total_loss‚85.72625057775451 _runtime‚ 95.97935884 /loss/FRt2/assignment_loss‚5.363601278398455 1loss/SRqqt2/assignment_loss‚3.828344828051949 0loss/SRqqt1/detection_loss‚3.581411051057594 .loss/FBt1/detection_loss‚51.91526733062886 epoch‚0 trainer/global_step‚149 _step‚11 0loss/FRt1/assignment_loss‚4.5797628160933375 .loss/FBt2/detection_loss‚7.884288258142944 % _timestamp‚1.7842381541157153e+09 2loss/SRqqt1/assignment_loss‚3.8956438702596743 0loss/FBt1/assignment_loss‚0.8923708421212656 0loss/FBt2/assignment_loss‚0.4664016754305276 0loss/FRt1/detection_loss‚0.45222057356272866 "mà·Ï‚(:’ Ó åÒäùôA gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚210.988' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.32933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ußq]KЂ(:À Ó åÒ•õ†B% disk./.usageGB‚1138.5554695129395 disk.nvme1n1.in‚ 0.0390625$ disk.nvme1c1n1.out‚ 13.7548828125* proc.memory.percent‚17.149829864501953 network.recv‚67176 disk.md0.out‚ 2.59765625 disk.dm-0.in‚0.078125 disk.dm-1.out‚ 1.107421875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 0.0390625( proc.memory.availableMB‚ 2077.6484375! proc.memory.rssMB‚ 2809.828125( disk./.usagePercent‚74.1609164313916" disk.nvme1n1.out‚ 13.7548828125 disk.md0.in‚0 disk.dm-0.out‚160.0146484375 disk.nvme0n1.in‚ 0.0390625" disk.nvme0n1.out‚ 152.951171875 cpu‚25.540352129774988 network.sent‚158642 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0390625$ disk.nvme0c0n1.out‚ 152.951171875$ memory_percent‚87.31904029846191 proc.cpu.threads‚16S’Ï>KÑ‚(:À â åÒÛ¡ÿA network.sent‚205814) disk./.usagePercent‚74.16097010177813% disk./.usageGB‚1138.5562934875488 disk.nvme0n1.in‚ 0.0390625$ disk.nvme0n1.out‚163.77490234375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚163.77490234375 network.recv‚71874 disk.nvme1c1n1.in‚ 0.0390625" disk.nvme1n1.out‚ 14.8056640625! disk.dm-0.out‚171.36376953125 disk.nvme0c0n1.in‚ 0.0390625* proc.memory.percent‚17.149829864501953 disk.md0.in‚0 disk.dm-0.in‚0.078125 disk.dm-1.in‚0% proc.memory.availableMB‚ 2077.0625! proc.memory.rssMB‚ 2809.828125 proc.cpu.threads‚16$ disk.nvme1c1n1.out‚ 14.8056640625 disk.nvme1n1.in‚ 0.0390625 disk.md0.out‚2.796875 disk.dm-1.out‚ 1.18359375$ memory_percent‚87.32261657714844 cpu‚25.5489807286307355ºæ$šÒ‚(: â åÒˆÔïB gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.05' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.01666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•…ö.0Ó+ _wandbruntime‚126  _runtime‚126Ñu‘æÆÔÀ ' lr-AdamW/pg1‚0.00011774199722953189 ' lr-AdamW/pg2‚0.00011774199722953189 trainer/global_step‚199 % _timestamp‚1.7842381849656386e+09 _runtime‚ 126.829628149 _step‚12_Ã>ñÊÕÄ % _timestamp‚1.7842381849656386e+09 _runtime‚ 126.829628149 _step‚12 ' lr-AdamW/pg1‚0.00011774199722953189 ' lr-AdamW/pg2‚0.00011774199722953189 trainer/global_step‚199 %¥90Ö+ _wandbruntime‚126  _runtime‚126;ÃVÔ¯'שO OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1978632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15049102458368 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1968840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14876542738672 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1529773 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1457921 _runtime‚ 126.830493041 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15207164016640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1529771 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11272897800704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 158150038528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15207269536256 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚445593 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 163351478272 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15034245814924 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 157616291354 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1529773 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1529771 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚13 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15207269536256 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15049102458368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1012501 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11272803691008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚445424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11436155169280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 158061558272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15049102458368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1012501 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11272803691008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚445424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11436155169280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 158061558272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15049102458368 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1968840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14876542738672 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1529773 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1457921 _runtime‚ 126.830493041 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15207164016640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1529771 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11272897800704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 158150038528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15207269536256 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚445593 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 163351478272 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15034245814924 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 157616291354 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1529773 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1529771 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚13 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15207269536256 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14950816224704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842381853622627e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 158935761408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11493097199616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11328941419008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1017553 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15124252124672 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚447658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225823232 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 158399785954 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1978660 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1988466 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131713536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 164287494144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 158847273984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15109216010658 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14950833264064 _runtime‚ 127.226420885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 164155780608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15283099398656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1465211 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682௫M½&Û·M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11493097199616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11328941419008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1017553 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15124252124672 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚447658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225823232 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 158399785954 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1978660 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1988466 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131713536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 164287494144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 158847273984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15109216010658 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14950833264064 _runtime‚ 127.226420885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 164155780608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15283099398656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1465211 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1978660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1017557 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14950816224704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842381853622627e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 158935761408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812‹T'¶0Ü+ _wandbruntime‚127  _runtime‚127]5„!þÝø 1loss/FBt1/assignment_loss‚0.43636625099157833 0loss/FBt2/assignment_loss‚0.3047800075439813 .loss/FBt1/detection_loss‚29.67306748194539 epoch‚0 trainer/global_step‚199 % _timestamp‚1.7842381853632398e+09 _runtime‚ 127.226895366 .loss/FRt1/assignment_loss‚4.51146521862973 1loss/SRqqt2/assignment_loss‚2.540174979135531 /loss/FRt2/detection_loss‚1.5260218229297906 1loss/SRqqt2/detection_loss‚1.3251921112765914 .loss/FBt2/detection_loss‚6.950590579325404 %loss/total_loss‚59.11217992932327 /loss/FRt2/assignment_loss‚5.337573120692005 2loss/SRqqt1/assignment_loss‚2.6942553657619683 0loss/FRt1/detection_loss‚0.36668999653078715 1loss/SRqqt1/detection_loss‚3.4460029945605144 _step‚15†½.‘Þü .loss/FRt1/assignment_loss‚4.51146521862973 1loss/SRqqt2/assignment_loss‚2.540174979135531 /loss/FRt2/detection_loss‚1.5260218229297906 1loss/SRqqt2/detection_loss‚1.3251921112765914 .loss/FBt2/detection_loss‚6.950590579325404 %loss/total_loss‚59.11217992932327 /loss/FRt2/assignment_loss‚5.337573120692005 2loss/SRqqt1/assignment_loss‚2.6942553657619683 0loss/FRt1/detection_loss‚0.36668999653078715 1loss/SRqqt1/detection_loss‚3.4460029945605144 _step‚15 1loss/FBt1/assignment_loss‚0.43636625099157833 0loss/FBt2/assignment_loss‚0.3047800075439813 .loss/FBt1/detection_loss‚29.67306748194539 epoch‚0 trainer/global_step‚199 % _timestamp‚1.7842381853632398e+09 _runtime‚ 127.226895366¤B¤8Iß‚(:¾ ñ åÒ×âB disk.nvme0n1.in‚ 0.0390625 disk.md0.out‚3.078125 disk.dm-0.in‚0.078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0390625# memory_percent‚87.3234510421753 network.sent‚210586$ disk.nvme0n1.out‚171.16650390625 disk.nvme1c1n1.in‚ 0.0390625 disk.dm-1.out‚1.265625& disk.nvme0c0n1.out‚171.16650390625) proc.memory.availableMB‚ 2076.92578125! proc.memory.rssMB‚ 2809.828125* proc.memory.percent‚17.149829864501953# disk.nvme1c1n1.out‚ 15.591796875 disk.nvme1n1.in‚ 0.0390625! disk.nvme1n1.out‚ 15.591796875 disk.dm-1.in‚0) disk./.usagePercent‚74.16112887667165% disk./.usageGB‚1138.5587310791016 disk.md0.in‚0! disk.dm-0.out‚178.84033203125 cpu‚25.55241404722408 proc.cpu.threads‚16 network.recv‚75857uÙà‚(:… ñ åÒ¨°ÿA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á bá‚(:… €¡åÒùÏçA gpu.0.gpu‚92.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚219.78' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.26 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'm(PRâ‚(:Ç €¡åÒ‘×öA network.recv‚80565 disk.dm-1.out‚ 1.38720703125 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0390625# disk.nvme1n1.out‚16.23974609375# proc.memory.rssMB‚ 2809.83203125 cpu‚25.559922796267607 proc.cpu.threads‚16 network.sent‚256610$ disk./.usageGB‚1138.559268951416 disk.nvme0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625 disk.md0.in‚0 disk.md0.out‚ 3.36328125$ disk.nvme0c0n1.out‚ 172.853515625" disk.nvme0n1.out‚ 172.853515625 disk.dm-0.in‚0.078125 disk.dm-0.out‚180.4306640625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚87.32397556304932) disk./.usagePercent‚74.16116391150732% disk.nvme1c1n1.out‚16.23974609375) proc.memory.availableMB‚ 2076.83984375* proc.memory.percent‚17.149853706359863ouX0ã+ _wandbruntime‚158  _runtime‚158òO;Ää¾ & lr-AdamW/pg1‚0.0001473254136188615 & lr-AdamW/pg2‚0.0001473254136188615 trainer/global_step‚249 % _timestamp‚1.7842382161524112e+09 _runtime‚ 158.016386876 _step‚16®+Èå & lr-AdamW/pg1‚0.0001473254136188615 & lr-AdamW/pg2‚0.0001473254136188615 trainer/global_step‚249 % _timestamp‚1.7842382161524112e+09 _runtime‚ 158.016386876 _step‚16ßU´‡0æ+ _wandbruntime‚158  _runtime‚158Z…¯'ç©O SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1911223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18787906791074 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 196854822162 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1911221 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14079690091008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 _step‚17 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 203914815488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842382161533184e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18591069008272 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18806585773568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1265297 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 196941607060 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2461692 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 197411498496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19003997272064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 197499978752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1819190 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18788010615332 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127526400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19004102791680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2471484 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18806602812928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 204042341888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14079784200704 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 197499978752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1819365 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18591051968912 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 197411498496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _runtime‚ 158.017151217 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048X,°è­O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18806585773568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1265297 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 196941607060 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2461692 TGDeviceStatsMonitor.oFâF¥%n_train_batch_start/active_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 197411498496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19003997272064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 197499978752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1819190 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18788010615332 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127526400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19004102791680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2471484 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18806602812928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 204042341888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14079784200704 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 197499978752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1819365 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18591051968912 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 197411498496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _runtime‚ 158.017151217 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1911223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18787906791074 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 196854822162 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1911221 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14079690091008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 _step‚17 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 203914815488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842382161533184e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18591069008272 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 143654912004€¿0é+ _wandbruntime‚158  _runtime‚158rª:ʹ&ê³M B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390362 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19079932654080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 198285701632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 198285701632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18862980811066 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1270357 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1918852 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1918850 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1918852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚163 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18863084635436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19080038180864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213240320 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 204729611776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚556218 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18881735439872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2481318 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2471512 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18665342494304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18881735439872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1918850 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1826575 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2471512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚18 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1826408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19080038180864 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2481318 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 204848742400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18881752479232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18881752479232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14135827819008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14135921928704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18665359533664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842382165479202e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 198197214208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 197725101772 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 197638316762 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 198197214208 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1270353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 trainer/global_step‚249= ŸÂ½&ë·M bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14135921928704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18665359533664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842382165479202e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 198197214208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 197725101772 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 197638316762 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 198197214208 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1270353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 trainer/global_step‚249 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390362 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19079932654080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 198285701632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 198285701632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18862980811066 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1270357 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1918852 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1918850 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1918852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚163 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18863084635436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19080038180864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213240320 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 204729611776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚556218 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18881735439872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2481318 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2471512 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18665342494304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18881735439872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1918850 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1826575 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2471512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚18 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1826408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19080038180864 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2481318 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 204848742400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18881752479232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18881752479232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14135827819008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960¿l¨0ì+ _wandbruntime‚158  _runtime‚1580Œ2líü 2loss/SRqqt2/assignment_loss‚2.2969747510509215 .loss/FBt2/detection_loss‚6.688853140764063 2loss/SRqqt1/assignment_loss‚2.2337661820772454 1loss/FBt1/assignment_loss‚0.37660514066765943 0loss/FBt2/assignment_loss‚0.2527984700768932 % _timestamp‚1.7842382165489252e+09 _runtime‚ 158.412578772 _step‚19 /loss/FRt1/assignment_loss‚4.577946571983264 /loss/FBt1/detection_loss‚25.315729869099762 &loss/total_loss‚52.709114920598076 trainer/global_step‚249 epoch‚0 0loss/FRt2/assignment_loss‚5.0109491936148025 /loss/FRt1/detection_loss‚0.4828624343294337 /loss/FRt2/detection_loss‚1.4995107145227289 1loss/SRqqt1/detection_loss‚2.8724163628568227 1loss/SRqqt2/detection_loss‚1.10070208955448173ò=ðî€ /loss/FRt2/detection_loss‚1.4995107145227289 1loss/SRqqt1/detection_loss‚2.8724163628568227 1loss/SRqqt2/detection_loss‚1.1007020895544817 2loss/SRqqt2/assignment_loss‚2.2969747510509215 .loss/FBt2/detection_loss‚6.688853140764063 2loss/SRqqt1/assignment_loss‚2.2337661820772454 1loss/FBt1/assignment_loss‚0.37660514066765943 0loss/FBt2/assignment_loss‚0.2527984700768932 % _timestamp‚1.7842382165489252e+09 _runtime‚ 158.412578772 _step‚19 /loss/FRt1/assignment_loss‚4.577946571983264 /loss/FBt1/detection_loss‚25.315729869099762 &loss/total_loss‚52.709114920598076 trainer/global_step‚249 epoch‚0 0loss/FRt2/assignment_loss‚5.0109491936148025 /loss/FRt1/detection_loss‚0.4828624343294337Ë’žï‚(:“ ¡åÒä²ïA gpu.0.gpu‚92.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚138.489' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚46.163000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯®îÒOð‚(:Ä ¡åÒÀŽÿA disk.nvme1c1n1.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0390625 disk.md0.out‚ 3.6484375 network.sent‚263504( disk./.usagePercent‚73.9842017142242 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.0390625# disk.nvme1n1.out‚16.93115234375* proc.memory.percent‚17.149853706359863 network.recv‚84673 disk.nvme0c0n1.in‚ 0.0390625" disk.nvme0n1.out‚ 188.052734375% disk.nvme1c1n1.out‚16.93115234375 disk.dm-0.out‚195.6142578125 disk.dm-1.in‚0 disk.dm-1.out‚ 1.47705078125$ memory_percent‚87.32078075408936 disk.rbd0.out‚0 disk.md0.in‚0 disk.dm-0.in‚0.078125) proc.memory.availableMB‚ 2077.36328125# proc.memory.rssMB‚ 2809.83203125 cpu‚25.56327938232255 proc.cpu.threads‚16$ disk./.usageGB‚1135.842456817627$ disk.nvme0c0n1.out‚ 188.052734375`šæ)Jñ‚(:Ä ž¡åÒ³¼ÖA disk.nvme0c0n1.in‚ 0.0390625 disk.nvme0n1.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0390625# disk.nvme1n1.out‚17.52685546875! disk.dm-0.out‚200.43798828125 disk.dm-1.out‚ 1.51416015625" proc.memory.rssMB‚ 2809.8359375% disk./.usageGB‚1135.8427200317383 disk.rbd0.out‚0# memoryø‚Ö_percent‚87.3220443725586* proc.memory.percent‚17.149877548217773& disk.nvme0c0n1.out‚192.73974609375$ disk.nvme0n1.out‚192.73974609375% disk.nvme1c1n1.out‚17.52685546875 disk.md0.in‚0& proc.memory.availableMB‚ 2077.15625 cpu‚25.566221288524606 proc.cpu.threads‚16 network.sent‚310918 network.recv‚89545' disk./.usagePercent‚73.984218858931 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 0.0390625 disk.md0.out‚3.828125 disk.dm-0.in‚0.078125 disk.dm-1.in‚0Óå9Xò‚(:… ž¡åÒû¦¥B gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.098' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tg0ó+ _wandbruntime‚189  _runtime‚189»4“þÅô¿ trainer/global_step‚299 $ _timestamp‚1.784238247273319e+09 _runtime‚ 189.137275302 _step‚20 ' lr-AdamW/pg1‚0.00017690883000819115 ' lr-AdamW/pg2‚0.00017690883000819115T"_Éõà _step‚20 ' lr-AdamW/pg1‚0.00017690883000819115 ' lr-AdamW/pg2‚0.00017690883000819115 trainer/global_step‚299 $ _timestamp‚1.784238247273319e+09 _runtime‚ 189.137275302ä4‹0ö+ _wandbruntime‚189  _runtime‚189½“È®'÷¨O OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2292671 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2964336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16886576491008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 236849918976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2179947 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784238247274192e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129623552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2954544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2954544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 trainer/global_step‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22541671591482 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2180133 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _runtime‚ 189.138028723 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22800936047104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22800830527488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16886670600704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 244474644480 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 236180137868 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2292671 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2964336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17131274868736 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 236093352970 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22541775415740 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 236849918976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 236761438720 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5257009 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22800830527488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223733248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17131051135488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚661850 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2292673 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22305595277872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22305578238512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2292673 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22564069088768 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2954544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 trainer/global_step‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22541671591482 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2180133 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _runtime‚ 189.138028723 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22800936047104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22800830527488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16886670600704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 244474644480 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 236180137868 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2292671 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2964336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17131274868736 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 236093352970 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22541775415740 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 236849918976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 236761438720 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5257009 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22800830527488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223733248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17131051135488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚661850 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2292673 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22305595277872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22305578238512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2292673 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22564069088768 H6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2964364 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚663976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1523157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22876871436288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22616745611474 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2964364 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22639235794432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16942808328704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245285533184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚22 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22639218755072 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2974170 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22876765909504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22876871436288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 237547154432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22616849435844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 236963632580 trainer/global_step‚299 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2300302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22639218755072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16942714219008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 237635641856 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2974170 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17187999752192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2300300 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2187129 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245415149568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812n?èû¶M E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2300300 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 237635641856 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2187304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2964364 H@a ³–Ô"DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚663976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1523157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22876871436288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22616745611474 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2964364 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22639235794432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16942808328704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245285533184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚22 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22639218755072 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2974170 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22876765909504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22876871436288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 237547154432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22616849435844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 236963632580 trainer/global_step‚299 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2300302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22639218755072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16942714219008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 237635641856 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2974170 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17187999752192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2300300 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2187129 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245415149568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 237547154432„uS*0ü+ _wandbruntime‚189  _runtime‚189F˜…Úýû 2loss/SRqqt2/assignment_loss‚1.7176257449684413 .loss/FBt2/detection_loss‚5.917715932798419 &loss/total_loss‚45.694102487040176 /loss/FRt2/assignment_loss‚4.117311813841305 0loss/FRt1/detection_loss‚0.41512695573197306 /loss/FRt2/detection_loss‚1.4975496540515005 trainer/global_step‚299 _runtime‚ 189.533189617 _step‚23 1loss/FBt1/assignment_loss‚0.25789991968194953 1loss/FBt2/assignment_loss‚0.15969106406870148 epoch‚0 $ _timestamp‚1.784238247669572e+09 /loss/FRt1/assignment_loss‚4.501588198891559 2loss/SRqqt1/assignment_loss‚1.9236007479903026 0loss/SRqqt1/detection_loss‚2.934904344386199 1loss/SRqqt2/detection_loss‚1.1188745026095235 /loss/FBt1/detection_loss‚21.132213608020297K{qCþÿ .loss/FBt2/detection_loss‚5.917715932798419 &loss/total_loss‚45.694102487040176 /loss/FRt2/assignment_loss‚4.117311813841305 0loss/FRt1/detection_loss‚0.41512695573197306 /loss/FRt2/detection_loss‚1.4975496540515005 trainer/global_step‚299 _runtime‚ 189.533189617 _step‚23 1loss/FBt1/assignment_loss‚0.25789991968194953 1loss/FBt2/assignment_loss‚0.15969106406870148 epoch‚0 $ _timestamp‚1.784238247669572e+09 /loss/FRt1/assignment_loss‚4.501588198891559 2loss/SRqqt1/assignment_loss‚1.9236007479903026 0loss/SRqqt1/detection_loss‚2.934904344386199 1loss/SRqqt2/detection_loss‚1.1188745026095235 /loss/FBt1/detection_loss‚21.132213608020297 2loss/SRqqt2/assignment_loss‚1.7176257449684413±)JiMÿ‚(: ­¡åÒ¸•B' proc.memory.availableMB‚ 2077.046875 proc.cpu.threads‚16 network.recv‚93679) disk./.usagePercent‚73.98421960435304 disk.dm-0.in‚0.078125! disk.dm-0.out‚212.56689453125 disk.dm-1.out‚ 1.63916015625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚204.9736328125 disk.nvme0n1.in‚ 0.0390625 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 18.2958984375 disk.nvme1n1.in‚ 0.0390625" disk.nvme1n1.out‚ 18.2958984375 disk.md0.out‚4.171875$ memory_percent‚87.32271194458008" proc.memory.rssMB‚ 2809.8359375* proc.memory.percent‚17.149877548217773# disk./.usageGB‚1135.84273147583 disk.dm-1.in‚0 disk.nvme1c1n1.in‚ 0.0390625 cpu‚25.569747246099634 network.sent‚315673 disk.nvme0c0n1.in‚ 0.0390625# disk.nvme0n1.out‚204.9736328125 disk.md0.in‚0ŠØ€‚(:’ ­¡åÒø¨ÿA gpu.0.gpu‚74.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚170.338' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚56.77933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Ï@›‚(: ¼¡åÒå¸ìA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚199.673' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.55766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÛ‚[K‚‚(:À ¼¡åÒÙæüA disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme0n1.in‚ 0.0390625# disk.nvme0n1.out‚214.0947265625% disk.nvme1c1n1.out‚19.71044921875 disk.md0.in‚0 disk.dm-0.in‚0.078125 disk.dm-1.in‚0 disk.dm-1.out‚ 1.73291015625" memory_percent‚87.326979637146 disk.nvme1n1.in‚ 0.0390625% disk.nvme0c0n1.out‚214.0947265625) proc.memory.availableMB‚ 2076.34765625" proc.memory.rssMB‚ 2809.8359375* proc.memory.percent‚17.149877548217773 proc.cpu.threads‚16) disk./.usagePercent‚73.98417090344674 disk.md0.out‚ 4.68359375 disk.dm-0.out‚ 221.921875 cpu‚25.5729336631351 network.sent‚361683 network.recv‚98322$ disk./.usageGB‚1135.841983795166 disk.nvme1c1n1.in‚ 0.0390625# disk.nvme1n1.out‚19.71044921875Ûkù0ƒ+ _wandbruntime‚220  _runtime‚220É4=¿Æ„À _step‚24 ' lr-AdamW/pg1‚0.00020649224639752076 ' lr-AdamW/pg2‚0.00020649224639752076 trainer/global_step‚349 % _timestamp‚1.7842382784591186e+09 _runtime‚ 220.323072841³j9Ê…Ä trainer/global_step‚349 % _timestamp‚1.7842382784591186e+09 _runtime‚ 220.323072841 _step‚24 ' lr-AdamW/pg1‚0.00020649224639752076 ' lr-AdamW/pg2‚0.00020649224639752076k¡‡î0†+ _wandbruntime‚220  _runtime‚220¬``¯'‡©O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26295540216148 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 284997233664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26295436391890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26321569443328 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2674123 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2540113 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3447396 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19978681760768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842382784600039e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26020104508112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6121517 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2540295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26597663782912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 _runtime‚ 220.327335712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26321569443328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1770901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19693557000704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26321552403968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3457188 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 276199859200 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3447396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚769394 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 275331883778 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _step‚25 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26597663782912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 285124760064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1770897 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚769216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19693462891008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2674121 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2674123 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19978460124672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26020104508112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6121517 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2540295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26597663782912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 _runtime‚ 220.327335712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26321569443328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1770901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19693557000704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26321552403968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3457188 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 276199859200 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3447396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚769394 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 275331883778 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _step‚25 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26597663782912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 285124760064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pooG‡ºŽl.freed‚1770897 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚769216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19693462891008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2674121 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2674123 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19978460124672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0»À³Ö0‰+ _wandbruntime‚220  _runtime‚220'Ç1¸&Š²M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚771347 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20035636055552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _step‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20035406038016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26094395033504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26673704691712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3467022 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2547487 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2681752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276985582080 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 276897094656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 276115378378 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1775957 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26370510411882 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2681752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2547300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚771530 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 276202163388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6138966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2681750 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19749600619008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚349 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26396702070272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26396719109632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19749694728704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 285805419008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3457216ªô·¼&‹¶M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _step‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20035406038016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26094395033504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26673704691712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3467022 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2547487 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2681752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276985582080 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 276897094656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 276115378378 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1775957 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26370510411882 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2681752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2547300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚771530 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 276202163388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6138966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2681750 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19749600619008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚349 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26396702070272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26396719109632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19749694728704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 285805419008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3457216 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚771347 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20035636055552 à_Á0Œ+ _wandbruntime‚220  _runtime‚220lèêðý÷ trainer/global_step‚349 _runtime‚ 220.718858314 _step‚27 0loss/FRt2/assignment_loss‚3.6828305483529427 1loss/SRqqt2/assignment_loss‚1.611593817120774 1loss/FBt1/assignment_loss‚0.20464643880750064 .loss/FRt1/detection_loss‚0.480861824629822 0loss/SRqqt1/detection_loss‚4.019393273229097 .loss/FBt2/detection_loss‚5.268699081121132 %loss/total_loss‚43.87260572924207 /loss/FRt1/assignment_loss‚4.065478253913023 /loss/FBt1/detection_loss‚20.121387925287078 epoch‚0 2loss/SRqqt1/assignment_loss‚1.8142209143414825 1loss/FBt2/assignment_loss‚0.12150545895962171 /loss/FRt2/detection_loss‚1.4493817137090736 0loss/SRqqt2/detection_loss‚1.032606479770522 $ _timestamp‚1.784238278855241e+09N£S£Žû 0loss/FRt2/assignment_loss‚3.6828305483529427 1loss/SRqqt2/assignment_loss‚1.611593817120774 1loss/FBt1/assignment_loss‚0.20464643880750064 .loss/FRt1/detection_loss‚0.480861824629822 0loss/SRqqt1/detection_loss‚4.019393273229097 .loss/FBt2/detection_loss‚5.268699081121132 %loss/total_loss‚43.87260572924207 /loss/FRt1/assignment_loss‚4.065478253913023 /loss/FBt1/detection_loss‚20.121387925287078 epoch‚0 2loss/SRqqt1/assignment_loss‚1.8142209143414825 1loss/FBt2/assignment_loss‚0.12150545895962171 /loss/FRt2/detection_loss‚1.4493817137090736 0loss/SRqqt2/detection_loss‚1.032606479770522 $ _timestamp‚1.784238278855241e+09 trainer/global_step‚349 _runtime‚ 220.718858314 _step‚27‹S‚(:’ Ë¡åÒ·´ûA gpu.0.gpu‚41.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.367' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.45566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ[ôE‚(:º Ë¡åÒŠ´‹B network.recv‚101713) disk./.usagePercent‚73.98416891565465 disk.nvme0n1.out‚ 227.84375" proc.memory.rssMB‚ 2809.8359375* proc.memory.percent‚17.149877548217773 cpu‚25.57356267569769 network.sent‚368361 disk.dm-1.in‚0 disk.dm-1.out‚ 1.78759765625 disk.rbd0.out‚0 disk.nvme0c0n1.out‚ 227.84375 disk.nvme0n1.in‚ 0.0390625# disk.nvme1n1.out‚20.44775390625 disk.dm-0.out‚ 235.50390625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0390625$ memory_percent‚87.32931613922119) proc.memory.availableMB‚ 2075.96484375 proc.cpu.threads‚16% disk.nvme1c1n1.out‚20.44775390625$ disk./.usageGB‚1135.841953277588 disk.nvme0c0n1.in‚ 0.0390625 disk.md0.in‚0 disk.md0.out‚5.078125 disk.dm-0.in‚0.078125è$R‘‚(:Ç Ú¡åÒŠîA% disk.nvme1c1n1.out‚21.33349609375 disk.nvme1n1.in‚ 0.0390625 disk.dm-1.out‚ 1.8896484375 cpu‚25.577154828934766) disk./.usagePercent‚73.98419624779594$ disk./.usageGB‚1135.842372894287 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625 disk.dm-0.in‚0.078125$ memory_percent‚87.33224868774414' proc.memory.availableMB‚ 2075.484375" proc.memory.rssMB‚ 2809.8359375 disk.rbd0.in‚0# disk.nvme1n1.out‚21.33349609375 disk.dm-1.in‚0 proc.cpu.threads‚16 network.sent‚413572 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.0390625$ disk.nvme0n1.out‚230.06396484375 disk.md0.in‚0 disk.md0.out‚ 5.3359375 disk.dm-0.out‚237.9482421875* proc.memory.percent‚17.149877548217773 network.recv‚105954& disk.nvme0c0n1.out‚230.06396484375Ýp<’‚(:’ Ú¡åÒí® B gpu.0.gpu‚58.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.353' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.78433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T¨®Ö0“+ _wandbruntime‚251  _runtime‚251s°Ü”¼ & lr-AdamW/pg1‚0.0002360756627868504 & lr-AdamW/pg2‚0.0002360756627868504 trainer/global_step‚399 # _timestamp‚1.78423830995285e+09 _runtime‚ 251.818402272 _step‚28L6ooÆ•À & lr-AdamW/pg2‚0.0002360756627868504 trainer/global_step‚399 # _timestamp‚1.78423830995285e+09 _runtime‚ 251.818402272 _step‚28 & lr-AdamW/pg1‚0.0002360756627868504äˆBÁ0–+ _wandbruntime‚251  _runtime‚251ƒlŽw—©O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219538944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30394602557952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22500443400704 % _timestamp‚1.7842383099534333e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2899820 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 251.818507083 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3055573 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125429248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325523120640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3940248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22500349291008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29734630777712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30079035719168 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 315549799424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30394497038336 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3940248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3950040 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3055571 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6995819 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29734647817072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30394602557952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 315461319168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22825872411648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30049201192298 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 314657199484œM|³'˜­O [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30049201192298 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 314657199484 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219538944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30394602557952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22500443400704 % _timestamp‚1.7842383099534333e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2899820 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 251.818507083 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3055573 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125429248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325523120640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3940248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22500349291008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29734630777712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30079035719168 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 315549799424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30394497038336 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3940248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3950040 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3055571 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6995819 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29734647817072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30394602557952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 315461319168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22825872411648'>)Ã0™+ _wandbruntime‚252  _runtime‚252–•Ó¹&š³M YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 trainer/global_step‚399 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3063202 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 315353909186 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7013268 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3063202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22883041895424 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30124379036660 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30154185385472 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30470537947136 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 316247034880 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30470432420352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3959874 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3959874 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30154185385472 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚878289 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22882811877888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135907840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 316335522304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7013268 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 326324858880 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30124275212290 _step‚30 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 316335522304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29808938342464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3950068 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30154202424832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29808921303104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2028757 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 316247034880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842383103472488e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2907042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3063200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2907224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3950068 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30154202424832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22556581128704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 326460766720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3063200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22556487019008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 252.211405399 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30470537947136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7023076 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 316335522304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7013268 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 326324858880 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30124275212290 _step‚30 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 316335522304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29808938342464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3950068 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30154202424832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29808921303104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2028757 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 316247034880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842383103472488e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2907042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3063200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2907224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3950068 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30154202424832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22556581128704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 326460766720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3063200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22556487019008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 252.211405399 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30470537947136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reR–«!´served_bytes.small_pool.current‚ 272629760 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7023076 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 EQ`NõbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPœNþ÷JöbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP§}íJ÷bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP§yfJøbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPõªJùbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP€ÓX]NúbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÝ"-HûbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPøÚ×iJübE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœ0µ!HýbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPî1JþbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPíä™ ;ÿb6 jet/accuracy_1_of_2"trainer/global_step2:JPó³;€b6 jet/accuracy_1_of_5"trainer/global_step2:JP¬žösJbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPdº¶J‚bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœHƒbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP õîJ„bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³¿NH…bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP´` ¼J†bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPYÈ‹ J‡bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP8>qNˆbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPù·“°J‰bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¿"rNŠbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPüÝAjN‹bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP'²b)HŒbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP Ò×=HbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP·¢JŽbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP'i"dJbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÎÈî#JbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP{„ J‘bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÇÛ; H’bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP^o)ÀJ“bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP#. ;”b6 jet/accuracy_1_of_1"trainer/global_step2:JPò ÙÐJ•bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPåÛðN–bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPúH¨H—bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPX48J˜bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPDÖ¿J™bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP×gJšbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPkú?J›bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPŠ CTNœbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¯Âü,HbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP/'éˆNžbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP bu›NŸbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP&|vâJ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP§õñJ¡bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP7àÅJ¢bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPü›:J£bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPºÆ§ÝJ¤bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP<{CGJ¥bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPz®|ËJ¦bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP`ðŠ®H§bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP*ô$N¨bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP:{‚ðN©bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÇ:†HªbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPû¡ñnH«bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPèLŸvJ¬bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÌ óqH­bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¨.ÀÝN®bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPØŠN¯bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP;ÚH°bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPקø†H±bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPê‡ãžJ²bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÊ?h,N³bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP· !&N´bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP{³ NµbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP( d0@¶b; particle/accuracy_3_of_4"trainer/global_step2:JPu 8@·b; particle/accuracy_5_of_5"trainer/global_step2:JP4ÒN¸bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPõT[J¹bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPê\M#@ºb; particle/accuracy_1_of_3"trainer/global_step2:JP÷­Ò1J»bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP˜ºªãN¼bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP1‘lóH½bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP\o@H¾bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPi.øG¿bB validation_average_jet_accuracy"trainer/global_step2:JPà„M&NÀbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP©WçGNÁbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPW{7âJÂbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÎ x4HÃbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPƒQHÄbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP˜–_0JÅbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¥(¸HÆbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP~ø:@Çb; particle/accuracy_2_of_5"trainer/global_step2:JPB§6HÈbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP+·ŠNÉbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPêƒ4@HÊbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPƆTlNËbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPè8alHÌbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPZV‘5HÍbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP$pµ&JÎbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP (JÏbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP&»ÉÎJÐbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPª^"êJÑbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP;&M©HÒbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP(' -NÓbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPMTÆÖHÔbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP°$×sNÕbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP[u¦8JÖbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPà èoH×bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPêöÛhNØbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP1QJÙbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP:©-ŒHÚbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPu3P>JÛbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP½õcNÜbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP}ã3NÝbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPœ¸-üHÞbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPl´ý@ßb; particle/accuracy_2_of_4"trainer/global_step2:JP»aÔJàbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPD[éHábC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPµÂøHâbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¤r!NãbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPrMâ+;äb6 jet/accuracy_2_of_2"trainer/global_step2:JP•8–JåbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPöÓâ¬HæbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPñþt@çb; particle/accuracy_4_of_4"trainer/global_step2:JP¸€e-;èb6 jet/accuracy_5_of_5"trainer/global_step2:JP5U¢NébI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP¸Ã™yJêbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPYˆ‰JëbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¿¤‚:HìbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPâjò?HíbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÊÒHîbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPª’òŒHïbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPKe`›JðbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¬ s¾JñbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP²'[¿HòbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPGœ“JóbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPi¨ü/NôbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP#ò“NõbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP1-ŸôJöbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŸõ×N÷bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP*;ØJøbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâÝNùbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP(ã}JúbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP픞,HûbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP5>RuHübC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP<ê2UNýbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÐW@{NþbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPº]b›JÿbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPç9®ÌJ€bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPRž¥ÊHbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP ‹SJ‚bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP“•JƒbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPª³ H„bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPï§û;…b6 jet/accuracy_3_of_3"trainer/global_step2:JP^¤â\J†bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP†¯5J‡bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPë•ÚKJˆbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÆ*Q&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.18771088123321533 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970992803573608 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21154971420764923 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995473325252533 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.784238323629446e+09 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0347975417971611 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.10235040634870529 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9139668345451355 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5370370149612427 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1588624268770218 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.14160948991775513 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3781181573867798 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27642279863357544 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13592137396335602 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13060151040554047 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.09821219742298126 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_2_of_2‚0.05253668501973152 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20736131072044373 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9325625896453857 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.756706178188324 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2061481475830078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.03609639033675194 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.01948458142578602 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09507647156715393 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15486764907836914 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978028535842896 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7698750495910645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4267464280128479 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21807578206062317 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1920730471611023 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.030879249796271324 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9978632926940918 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.21321314573287964 ;"Purity/2FRt0SRqqt*FBt/event_purt‡òªÃ#ity‚0.004885269328951836 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2724256217479706 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1915205717086792 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 : Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0069444444961845875 ,jet/accuracy_3_of_3‚0.011538567952811718 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970992803573608 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.18001843988895416 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17887485027313232 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8685582280158997 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994015634059906 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1295463591814041 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02627112716436386 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09560656547546387 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14980702102184296 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09293648600578308 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932456016540527 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15690676867961884 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0027167261578142643 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977167248725891 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3232150077819824 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18721283972263336 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03203696385025978 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03530040383338928 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.017095981165766716 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9969245195388794 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25464358925819397 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005261601414531469 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1597299426794052 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.1619940996170044 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.17105691134929657 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15816524624824524 ,jet/accuracy_3_of_4‚0.013755329884588718 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0651041716337204 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.1504804641008377 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17887485027313232 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.5977213978767395 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.16716617345809937 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.05890337750315666 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 epoch‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002051272662356496 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20241200923919678 'particle/f_score‚0.5301441550254822 *loss/val_total_loss‚28.570858557641664 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3581795394420624 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7072945237159729 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5370370149612427 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879228472709656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.7967161536216736 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886248707771301 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10691044479608536 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20697557926177979 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5725962519645691 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00291073857806623 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07193317264318466 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18045268952846527 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20241200923919678 jet/accuracy_4_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 _step‚32 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.16549508273601532 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.996734619140625 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.19872868061065674 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.14462916553020477 )jet/accuracy_2_of_4‚0.100083127617836 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20532478392124176 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9919993281364441 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34606948494911194 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.18569539487361908 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03500000014901161 jet/accuracy_4_of_4‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.20687834918498993 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0794818252325058 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.004511812701821327 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.23660244047641754 *jet/accuracy_1_of_3‚0.6016249656677246 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.19675704836845398 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9709550142288208 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004625375848263502 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 +particle/sensitivity‚0.5340283513069153 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.27630287408828735 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1484375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.03543218597769737 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1826748102903366 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 _runtime‚ 265.49389706 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.4594230651855469 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10396473854780197 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20246709883213043 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0794818252325058 trainer/global_step‚399 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18906541168689728 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9854774475097656 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03203696385025978 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21914690732955933 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.986443281173706tßúÐÜ<ÃÖy 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945647716522217 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2598612904548645 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.023566845804452896 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.170719176530838 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14854443073272705 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997677743434906 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_4‚0.9951900243759155 particle/accuracy_5_of_5‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0031689838506281376 particle/accuracy_1_of_3‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002086323918774724 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9968242645263672 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 7validation_average_jet_accuracy‚0.15341761708259583 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.18771088123321533 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970992803573608 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21154971420764923 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995473325252533 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.784238323629446e+09 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0347975417971611 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.10235040634870529 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9139668345451355 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5370370149612427 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1588624268770218 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.14160948991775513 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3781181573867798 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27642279863357544 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13592137396335602 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13060151040554047 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.09821219742298126 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_2_of_2‚0.05253668501973152 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20736131072044373 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9325625896453857 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.756706178188324 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2061481475830078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.03609639033675194 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.01948458142578602 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09507647156715393 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15486764907836914 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978028535842896 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7698750495910645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4267464280128479 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21807578206062317 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1920730471611023 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.030879249796271324 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9978632926940918 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.21321314573287964 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004885269328951836 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2724256217479706 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1915205717086792 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 : Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0069444444961845875 ,jet/accuracy_3_of_3‚0.011538567952811718 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970992803573608 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.18001843988895416 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17887485027313232 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8685582280158997 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994015634059906 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1295463591814041 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02627112716436386 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09560656547546387 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14980702102184296 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09293648600578308 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932456016540527 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15690676867961884 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0027167261578142643 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977167248725891 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3232150077819824 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18721283972263336 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03203696385025978 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03530040383338928 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.017095981165766716 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9969245195388794 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25464358925819397 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005261601414531469 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1597299426794052 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.1619940996170044 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.17105691134929657 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15816524624824524 ,jet/accuracy_3_of_4‚0.013755329884588718 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0651041716337204 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.1504804641008377 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17887485027313232 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.5977213978767395 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.16716617345809937 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.05890337750315666 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 epoch‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002051272662356496 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20241200923919678 'particle/f_score‚0.5301441550254822 *loss/val_total_loss‚28.570858557641664 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3581795394420624 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7072945237159729 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5370370149612427 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879228472709656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.7967161536216736 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886248707771301 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10691044479608536 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20697557926177979 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5725962519645691 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00291073857806623 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07193317264318466 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18045268952846527 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20241200923919678 jet/accuracy_4_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 _step‚32 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.16549508273601532 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.996734619140625 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.19872868061065674 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.14462916553020477 )jet/accuracy_2_of_4‚0.100083127617836 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20532478392124176 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9919993281364441 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34606948494911194 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.18569539487361908 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03500000014901161 jet/accuracy_4_of_4‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.20687834918498993 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0794818252325058 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.004511812701821327 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.23660244047641754 *jet/accuracy_1_of_3‚0.6016249656677246 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.19675704836845398 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9709550142288208 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004625375848263502 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 +particle/sensitivity‚0.5340283513069153 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.27630287408828735 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1484375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.03543218597769737 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1826748102903366 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 _runtime‚ 265.49389706 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.4594230651855469 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10396473854780197 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20246709883213043 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0794818252325058 trainer/global_step‚399 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18906541168689728 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9854774475097656 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03203696385025978 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21914690732955933 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.986443281173706 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03090684674680233 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2013888955116272 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.11335403472185135 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.19891834259033203 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9965590238571167 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10314936935901642 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.15137943625450134 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0017361199716106057 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7836107611656189 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.16097916662693024 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4179976284503937 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.18345670402050018 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.09977993369102478 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.20016439259052277 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.07078029215335846 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 YŽE0VÄ‚(:Ë ø¡åÒ£õøA network.sent‚467322 disk.nvme1c1n1.in‚ 0.0390625% disk.nvme1n1.out‚1937.08935546875 disk.dm-0.in‚0.078125 disk.dm-1.in‚0 disk.dm-1.out‚ 2.09716796875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.0390625$ disk./.usageGB‚1138.863353729248 disk.nvme1n1.in‚ 0.0390625 disk.md0.in‚0! disk.dm-0.out‚2181.8564453125% disk.nvme0c0n1.out‚260.1611328125$ memory_percent‚88.31968307495117* proc.memory.percent‚17.356443405151367 cpu‚25.73236098336325 network.recv‚114972) disk./.usagePercent‚74.18097076888117 disk.nvme0n1.in‚ 0.0390625" proc.memory.rssMB‚ 2843.6796875 proc.cpu.threads‚16# disk.nvme0n1.out‚260.1611328125' disk.nvme1c1n1.out‚1937.08935546875 disk.md0.out‚ 6.14453125 disk.rbd0.out‚0' proc.memory.availableMB‚ 1913.703125-6-ÃÅ‚(:’ ø¡åÒâÈíA gpu.0.gpu‚72.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚212.243' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.74766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã~Ä ZÆ‚(:Ï ‡¢åÒ¦•ˆB network.sent‚524056 network.recv‚119756 disk.dm-1.in‚0! disk.dm-1.out‚129.89794921875% disk.nvme0c0n1.out‚908.5498046875 disk.nvme0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625$ memory_percent‚88.31987380981445# disk.nvme0n1.out‚908.5498046875% disk.nvme1n1.out‚2292.18505859375 disk.md0.in‚0' proc.memory.availableMB‚ 1913.671875* proc.memory.percent‚17.356514930725098 proc.cpu.threads‚16) disk./.usagePercent‚74.15052226653144% disk./.usageGB‚1138.3958930969238 disk.dm-0.in‚0.078125! disk.dm-0.out‚3056.7548828125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0390625' disk.nvme1c1n1.out‚2292.18505859375 cpu‚25.72651650311393 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.0390625 disk.md0.out‚ 6.50390625# proc.memory.rssMB‚ 2843.69140625?¤/h›Ç‚(: ‡¢åÒËÂüA gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚219.427' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.14233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Ü¡E0È+ _wandbruntime‚296  _runtime‚296*gMâÃɽ _runtime‚ 296.99920992 _step‚33 & lr-AdamW/pg1‚0.0002366636814256905 & lr-AdamW/pg2‚0.0002366636814256905 trainer/global_step‚449 % _timestamp‚1.7842383551352324e+09(§*ÇÇÊÁ % _timestamp‚1.7842383551352324e+09 _runtime‚ 296.99920992 _step‚33 & lr-AdamW/pg1‚0.0002366636814256905 & lr-AdamW/pg2‚0.0002366636814256905 trainer/global_step‚449!UqÏ^0Ë+ _wandbruntime‚297  _runtime‚297wöE‰BÌ®O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3413863 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 $ _timestamp‚1.784238355136113e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35203352514560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370240485888 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558821 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4515313 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110570496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34422624083504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 370240485888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204680192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3558821 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35203246816256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381736263168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2384756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558823 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 trainer/global_step‚449 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2384752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34833006330368 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3414052 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1029111 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34791952112077 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369224202795 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35203246816256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8084277 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 369310989213 _runtime‚ 296.999932922aNw]¸'ͲO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35203352514560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 370240485888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204680192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3558821 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35203246816256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381736263168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2384756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558823 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 trainer/global_step‚449 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2384752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34833006330368 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3414052 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1029111 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34791952112077 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369224202795 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35203246816256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8084277 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 369310989213 _runtime‚ 296.999932922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3413863 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 $ _timestamp‚1.784238355136113e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35203352514560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370240485888 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558821 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4515313 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110570496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34422624083504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 HDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3566450 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4525133 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210964480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35279182198272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4535288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1031407 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35279287903744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34908173036032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 371114867712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35279287903744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 371026201600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26166738624000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26166832733696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4525133 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3421219 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3421041 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2389812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4525133 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3421219 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3421041 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2389812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3566450 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4525133 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210964480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35279182198272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4535288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1031407 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35279287903744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34908173036032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 371114867712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35279287903744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocat¾:?ý ed‚ 272629760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 371026201600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26166738624000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26166832733696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38137150353104 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5008165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38545613086707 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940273 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409590426112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38545716912485 % _timestamp‚1.7842383864912965e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38590489645568 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 408462733603 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2637552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1137342 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2637556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118959104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚ 328.358350913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28917487296000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39000080071680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3774725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5018306 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29340068487168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 _step‚38À‹ê¹'ݳO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39000185769984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5008165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38545613086707 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940273 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409590426112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38545716912485 % _timestamp‚1.7842383864912965e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38590489645568 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 408462733603 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2637552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1137342 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2637556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118959104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚ 328.358350913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28917487296000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39000080071680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3774725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5018306 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29340068487168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 _step‚38 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 409679085056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38590489645568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39000185769984 trainer/global_step‚499 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1137173 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 422368122368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8948436 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5018306 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 409590426112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38137150353104 H6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38620790932589 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚39 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2642608 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 409246228203 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410376141824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29396810260992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410376141824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚ 328.75120474 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚499 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5017985 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5028140 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28973719133696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38665639311872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3947902 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3781953 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5028140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3947900 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 410464807936 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5017985 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3947902 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38211440878496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2642612 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1139507 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38620687106699 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3782119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110563328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0úcqÂ&à¼M UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚39 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2642608 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 409246228203 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410376141824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29396810260992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410376141824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚ 328.75120474 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚499 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5017985 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5028140 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28973719133696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38665639311872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3947902 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3781953 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5028140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3947900 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 410464807936 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5017985 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3947902 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38211440878496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2642612 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1139507 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38620687106699 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3782119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110563328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38620790932589'ZBâ¯0á+ _wandbruntime‚328  _runtime‚328Ii÷âû -loss/FBt1/detection_loss‚6.80197023473422 _step‚40 epoch‚1 0loss/FRt1/assignment_loss‚4.1129978337668645 1loss/FBt2/assignment_loss‚0.12332081946312558 /loss/FRt1/detection_loss‚0.4957823073817535 /loss/FRt2/detection_loss‚1.4382230121022097 0loss/SRqqt1/detection_loss‚2.465976964133648 trainer/global_step‚499 2loss/SRqqt1/assignment_loss‚1.6683856198405471 2loss/SRqqt2/assignment_loss‚1.3548501778308286 .loss/FBt2/detection_loss‚5.972724603268352 &loss/total_loss‚28.582604742377868 0loss/FRt2/assignment_loss‚2.9601684715570555 1loss/FBt1/assignment_loss‚0.18901277001120378 1loss/SRqqt2/detection_loss‚0.9991919282880591 % _timestamp‚1.7842383868879843e+09 _runtime‚ 328.751623161® z1ãÿ % _timestamp‚1.7842383868879843e+09 _runtime‚ 328.751623161 -loss/FBt1/detection_loss‚6.80197023473422 _step‚40 epoch‚1 0loss/FRt1/assignment_loss‚4.1129978337668645 1loss/FBt2/assignment_loss‚0.12332081946312558 /loss/FRt1/detection_loss‚0.4957823073817535 /loss/FRt2/detection_loss‚1.4382230121022097 0loss/SRqqt1/detection_loss‚2.465976964133648 trainer/global_step‚499 2loss/SRqqt1/assignment_loss‚1.6683856198405471 2loss/SRqqt2/assignment_loss‚1.3548501778308286 .loss/FBt2/detection_loss‚5.972724603268352 &loss/total_loss‚28.582604742377868 0loss/FRt2/assignment_loss‚2.9601684715570555 1loss/FBt1/assignment_loss‚0.18901277001120378 1loss/SRqqt2/detection_loss‚0.9991919282880591(ÀzÙä‚(:’ ´¢åÒÔÄËA gpu.0.gpu‚78.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚224.629' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.87633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÂu2[å‚(:Ð ´¢åÒ‰¡ÞA disk.rbd0.out‚0% disk.nvme0c0n1.out‚941.7548828125 disk.nvme0n1.in‚ 0.0390625' disk.nvme1c1n1.out‚2297.38232421875( disk./.usagePercent‚74.1505468654586 disk.dm-0.in‚0.078125! disk.dm-1.out‚131.48486328125 disk.nvme0c0n1.in‚ 0.0390625 disk.md0.in‚0 disk.md0.out‚ 7.38671875$ memory_percent‚89.10839557647705) proc.memory.availableMB‚ 1784.48046875 network.sent‚596372 network.recv‚132107 disk.dm-1.in‚0 disk.nvme1n1.in‚ 0.0390625% disk.nvme1n1.out‚2297.38232421875# proc.memory.rssMB‚ 2843.69140625* proc.memory.percent‚17.356514930725098 proc.cpu.threads‚16 disk.rbd0.in‚0# disk.nvme0n1.out‚941.7548828125 disk.nvme1c1n1.in‚ 0.0390625 cpu‚25.710700734183593% disk./.usageGB‚1138.3962707519531 disk.dm-0.out‚3091.623046875¡'Ý’æ‚(:‡ âåÒøáïA gpu.0.gpu‚92.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚122.559' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.853 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Â`’Yç‚(:Î âåÒ—”ÿA network.recv‚136668) disk./.usagePercent‚74.15088131147843 disk.nvme0c0n1.in‚ 0.0390625 disk.nvme1c1n1.in‚ 0.0390625& disk.nvme1c1n1.out‚2300.0087890625 disk.nvme1n1.in‚ 0.0390625" memory_percent‚89.107346534729) proc.memory.availableMB‚ 1784.65234375& disk.nvme0c0n1.out‚950.95361328125 disk.md0.in‚0 disk.md0.out‚ 7.87109375 disk.dm-0.in‚0.078125" disk.dm-0.out‚3102.09716796875 disk.dm-1.out‚ 131.80078125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2843.69140625% disk./.usageGB‚1138.4014053344727$ disk.nvme0n1.out‚950.95361328125 cpu‚25.706024589104953 network.sent‚659247 disk.nvme0n1.in‚ 0.0390625$ disk.nvme1n1.out‚2300.0087890625 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.356514930725098 proc.cpu.threads‚16žQ50è+ _wandbruntime‚359  _runtime‚359¤ÆéÀ _step‚41 ' lr-AdamW/pg1‚0.00023663358546300556 ' lr-AdamW/pg2‚0.00023663358546300556 trainer/global_step‚549 % _timestamp‚1.7842384178170886e+09 _runtime‚ 359.681048363?OÊêÄ ' lr-AdamW/pg1‚0.00023663358546300556 ' lr-AdamW/pg2‚0.00023663358546300556 trainer/global_step‚549 % _timestamp‚1.7842384178170886e+09 _runtime‚ 359.681048363 _step‚41) ƒÚŽ0ë+ _wandbruntime‚359  _runtime‚3590&²µ'ì¯O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42347972960768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31724373696000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4321723 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5511158 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449029025280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32187550488576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 462953238016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842384178179593e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42796913327104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42797019025408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447788050829 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42347972960768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130Œ2øwM í³O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42796913327104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5501017 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4321721 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32187326934016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9822738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 463082682880 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448940366336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2890356 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447788050829 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42347972960768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42347972960768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31724373696000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allŽòÜlocated_bytes.all.peak‚ 12301876224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4321723 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5511158 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449029025280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32187550488576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 462953238016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842384178179593e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42796913327104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42797019025408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚43 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42872954414592 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 360.077132759 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4142500 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42423139666432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42423122627072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32244283185664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4142666 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41925967148096 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9840187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121049088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329352 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5510837 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449726082048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1247092 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31780511424000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32244498344448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42872848709120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41925984187456 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42374451907107^'˜Ã&ò½M J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42374451907107 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚43 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42872954414592 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 360.077132759 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4142500 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42423139666432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42423122627072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32244283185664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4142666 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41925967148096 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9840187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121049088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329352 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5510837 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449726082048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1247092 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31780511424000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32244498344448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42872848709120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41925984187456+oFÎq0ó+ _wandbruntime‚360  _runtime‚360’™˜ýô÷ 2loss/SRqqt1/assignment_loss‚1.6874419538744132 1loss/SRqqt2/detection_loss‚0.8520445839196928 -loss/FBt1/detection_loss‚8.41900846687734 trainer/global_step‚549 % _timestamp‚1.7842384182142527e+09 2loss/SRqqt2/assignment_loss‚1.3650078230876794 .loss/FRt1/detection_loss‚0.461422765275813 /loss/FRt2/detection_loss‚1.2830251653973175 /loss/FBt2/detection_loss‚3.3181652729671463 %loss/total_loss‚29.12619879510485 _step‚44 0loss/FBt1/assignment_loss‚0.1224479333020501 1loss/FBt2/assignment_loss‚0.21823849317468977 0loss/SRqqt1/detection_loss‚4.565364733703351 0loss/FRt2/assignment_loss‚2.7955833185952286 epoch‚1 _runtime‚ 360.07786302 /loss/FRt1/assignment_loss‚4.038448284930132Îõ¶õû /loss/FRt1/assignment_loss‚4.038448284930132 2loss/SRqqt1/assignment_loss‚1.6874419538744132 1loss/SRqqt2/detection_loss‚0.8520445839196928 -loss/FBt1/detection_loss‚8.41900846687734 trainer/global_step‚549 % _timestamp‚1.7842384182142527e+09 2loss/SRqqt2/assignment_loss‚1.3650078230876794 .loss/FRt1/detection_loss‚0.461422765275813 /loss/FRt2/detection_loss‚1.2830251653973175 /loss/FBt2/detection_loss‚3.3181652729671463 %loss/total_loss‚29.12619879510485 _step‚44 0loss/FBt1/assignment_loss‚0.1224479333020501 1loss/FBt2/assignment_loss‚0.21823849317468977 0loss/SRqqt1/detection_loss‚4.565364733703351 0loss/FRt2/assignment_loss‚2.7955833185952286 epoch‚1 _runtime‚ 360.07786302,JÃàþžö‚(:“ á¢åÒ—ÙA gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.405' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.135000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉX(·_÷‚(:Ô á¢åÒ‘ïA disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2331.52880859375 disk.nvme0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625% disk.nvme1c1n1.out‚2308.498046875$ memory_percent‚89.10911083221436) proc.memory.availableMB‚ 1784.36328125# proc.memory.rssMB‚ 2843.69140625) disk./.usagePercent‚74.23753761697759 disk.md0.in‚0 disk.dm-1.out‚ 132.349609375% disk.nvme0n1.out‚2331.52880859375* proc.memory.percent‚17.356514930725098 cpu‚25.699133442447053 network.sent‚716047% disk./.usageGB‚1139.7317962646484 disk.md0.out‚ 10.453125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 2.89453125 proc.cpu.threads‚16 network.recv‚145567 disk.nvme1n1.in‚ 1.6640625# disk.nvme1n1.out‚2308.498046875 disk.dm-0.in‚ 4.55859375" disk.dm-0.out‚4485.33349609375„¥Ò‹’ø‚(:‡ ð¢åÒ¯ÄA gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.695' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M.âBbù‚(:× ð¢åÒ€“×A disk.nvme0n1.in‚ 2.89453125 network.sent‚768111) disk./.usagePercent‚74.23783951290189# memory_percent‚89.1080379486084( proc.memory.availableMB‚ 1784.5390625 cpu‚25.694849218179904 proc.cpu.threads‚16 network.recv‚150039 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.in‚0* proc.memory.percent‚17.356514930725098% disk./.usageGB‚1139.7364311218262 disk.nvme1n1.in‚ 1.6640625 disk.dm-0.in‚ 4.55859375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 2.89453125& disk.nvme0c0n1.out‚2354.4345703125$ disk.nvme0n1.out‚2354.4345703125# proc.memory.rssMB‚ 2843.69140625! disk.dm-1.out‚132.45654296875' disk.nvme1c1n1.out‚2310.30224609375% disk.nvme1n1.out‚2310.30224609375 disk.md0.out‚ 10.94921875! disk.dm-0.out‚4508.8779296875 disk.dm-1.in‚0 disk.rbd0.out‚0Nvš0ú+ _wandbruntime‚390  _runtime‚390bɉ]ÆûÀ ' lr-AdamW/pg1‚0.00023660713960219635 ' lr-AdamW/pg2‚0.00023660713960219635 trainer/global_step‚599 % _timestamp‚1.7842384487553644e+09 _runtime‚ 390.619322812 _step‚45(PÄ~güÄ ' lr-AdamW/pg1‚0.00023660713960219635 ' lr-AdamW/pg2‚0.00023660713960219635 trainer/gloЀæÄcbal_step‚599 % _timestamp‚1.7842384487553644e+09 _runtime‚ 390.619322812 _step‚45-4@„Û0ý+ _wandbruntime‚390  _runtime‚390Aä&i¸'þ²O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1352172 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 503648061440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 trainer/global_step‚599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6004010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10707183 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35034780809728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46593746582528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46053246513301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35035002267136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45566219931664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503520713728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 488378965504 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4495328 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46105473315328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703173 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 487026581637 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46593746582528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4495142 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46053142687523 _runtime‚ 390.62334182 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6004010 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46105473315328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221457408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34531354205696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1351990 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚46 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46593852280832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10697040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 488290306560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46593852280832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 488378965504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34531260096000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488@÷a¼'ÿ¶O OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6004010 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46105473315328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221457408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34531354205696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1351990 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚46 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46593852280832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10697040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 488290306560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46593852280832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 488378965504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34531260096000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1352172 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 503648061440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 trainer/global_step‚599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6004010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10707183 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35034780809728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46593746582528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46053246513301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35035002267136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45566219931664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503520713728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 488378965504 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4495328 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46105473315328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703173 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 487026581637 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46593746582528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4495142 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46053142687523 _runtime‚ 390.62334182 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0.>tö0€+ _wandbruntime‚391  _runtime‚391>Æ·fÂ&¼M _runtime‚ 391.015302347 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10724646 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 489076022272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46669787670016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 504330193408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3148208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46180622981632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚47 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487810076349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10714489 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6003689 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 489076022272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46128320533405 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4502428 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 $ _timestamp‚1.784238449151193e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4710802 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46180622981632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46669681964544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4710800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 504463825408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46180605942272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1354216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46669787670016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35091728017408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6013844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6013844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46180605942272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3148212 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35091955759104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133632000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34587491933696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34587397824000 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 487723289819 trainer/global_step‚599 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4710800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1354030 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 489164688384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45640510457056 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4710802 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46128216707515 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682$}F9Ö‚ÀM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46669787670016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 504330193408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3148208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46180622981632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚47 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487810076349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10714489 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6003689 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 489076022272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonit)º Ôðor.on_train_batch_end/requested_bytes.all.allocated‚46128320533405 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4502428 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 $ _timestamp‚1.784238449151193e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4710802 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46180622981632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46669681964544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4710800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 504463825408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46180605942272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1354216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46669787670016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35091728017408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6013844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6013844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46180605942272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3148212 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35091955759104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133632000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34587491933696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34587397824000 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 487723289819 trainer/global_step‚599 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4710800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1354030 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 489164688384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45640510457056 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4710802 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46128216707515 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _runtime‚ 391.015302347 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10724646 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 489076022272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712/‘iŽM0ƒ+ _wandbruntime‚391  _runtime‚391ƒ.•-ü„ö .loss/FRt2/detection_loss‚1.292551190994772 1loss/SRqqt2/detection_loss‚0.9436621076473497 $ _timestamp‚1.784238449152148e+09 .loss/FRt2/assignment_loss‚2.72582212223687 1loss/SRqqt1/assignment_loss‚1.600548028270714 0loss/FBt1/assignment_loss‚0.1472568919906104 1loss/SRqqt1/detection_loss‚2.8969159640407316 /loss/FBt1/detection_loss‚3.5095159179192508 /loss/FBt2/detection_loss‚7.7353942269941225 trainer/global_step‚599 /loss/FRt1/assignment_loss‚4.091301465731261 epoch‚1 _runtime‚ 391.015769568 %loss/total_loss‚26.95998746782152 1loss/SRqqt2/assignment_loss‚1.292158589964645 0loss/FRt1/detection_loss‚0.46774807581624783 _step‚48 0loss/FBt2/assignment_loss‚0.2571128862149433hM·®…ú 1loss/SRqqt2/assignment_loss‚1.292158589964645 0loss/FRt1/detection_loss‚0.46774807581624783 _step‚48 0loss/FBt2/assignment_loss‚0.2571128862149433 .loss/FRt2/detection_loss‚1.292551190994772 1loss/SRqqt2/detection_loss‚0.9436621076473497 $ _timestamp‚1.784238449152148e+09 .loss/FRt2/assignment_loss‚2.72582212223687 1loss/SRqqt1/assignment_loss‚1.600548028270714 0loss/FBt1/assignment_loss‚0.1472568919906104 1loss/SRqqt1/detection_loss‚2.8969159640407316 /loss/FBt1/detection_loss‚3.5095159179192508 /loss/FBt2/detection_loss‚7.7353942269941225 trainer/global_step‚599 /loss/FRt1/assignment_loss‚4.091301465731261 epoch‚1 _runtime‚ 391.015769568 %loss/total_loss‚26.959987467821520P6"b†‚(:’ ÿ¢åÒÜëÅA gpu.0.gpu‚92.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚116.165' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.72166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>u_‡‚(:Ô ÿ¢åÒ†öÑA network.sent‚773304! disk.dm-1.out‚132.56982421875 disk.nvme1c1n1.in‚ 1.6640625% disk.nvme1n1.out‚2311.14990234375 disk.md0.in‚0 disk.md0.out‚ 11.20703125# proc.memory.rssMB‚ 2843.69140625) disk./.usagePercent‚74.23780845365042$ disk./.usageGB‚1139.735954284668 disk.dm-0.in‚ 4.55859375 disk.nvme0c0n1.in‚ 2.89453125$ memory_percent‚89.10763263702393) proc.memory.availableMB‚ 1784.60546875* proc.memory.percent‚17.356514930725098 cpu‚25.692646855347682 network.recv‚154058 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚2362.740234375 disk.nvme0n1.in‚ 2.89453125# disk.nvme0n1.out‚2362.740234375 disk.dm-0.out‚ 4517.35546875 disk.rbd0.out‚0' disk.nvme1c1n1.out‚2311.14990234375 disk.nvme1n1.in‚ 1.6640625 proc.cpu.threads‚16b¼AŠœˆ‚(:‘ Ž£åÒ£íæA gpu.0.gpu‚30.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.517' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.50566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pv!Z‰‚(:Ï Ž£åÒÅöA% disk.nvme0c0n1.out‚2402.205078125# disk.nvme0n1.out‚2402.205078125$ disk.nvme1n1.out‚2327.6123046875 disk.dm-0.in‚ 4.55859375! disk.dm-0.out‚4572.7958984375 disk.rbd0.in‚0% disk./.usageGB‚1140.1585502624512 disk.nvme0c0n1.in‚ 2.89453125 disk.md0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚16 disk.nvme0n1.in‚ 2.89453125) proc.memory.availableMB‚ 1784.03515625# proc.memory.rssMB‚ 2843.69140625* proc.memory.percent‚17.356514930725098 cpu‚25.688825069710266 network.recv‚158678 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2327.6123046875 disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 11.36328125 disk.dm-1.in‚0 disk.dm-1.out‚ 132.71875$ memory_percent‚89.11111354827881 network.sent‚835694) disk./.usagePercent‚74.265334653147780çód0Š+ _wandbruntime‚421  _runtime‚421zòÄ‹¾ & lr-AdamW/pg1‚0.0002365730972571613 & lr-AdamW/pg2‚0.0002365730972571613 trainer/global_step‚649 % _timestamp‚1.7842384800859349e+09 _runtime‚ 421.949913693 _step‚49CtZ.Ȍ trainer/global_step‚649 % _timestamp‚1.7842384800859349e+09 _runtime‚ 421.949913693 _step‚49 & lr-AdamW/pg1‚0.0002365730972571613 & lr-AdamW/pg2‚0.00023657309725716131JÑí¡0+ _wandbruntime‚421  _runtime‚421 ïá4¹'Ž³O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527728905728 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4854852 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 526265112445 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5084623 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6486721 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 _step‚50 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 544215650816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49807011313709 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527640246784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚166 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37338146496000 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5084621 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37882234799104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 _runtime‚ 421.954354641 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49806907487931 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50390579837952 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50390685536256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11581485 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526178326027 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37882456256512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49280746201264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49280729161904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49862956630528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 527728905728 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496862 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1458900 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5084623 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527640246784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 544215650816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49807011313709 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527640246784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚166 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37338146496000 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5084621 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37882234799104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 _runtime‚ 421.954354641 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49806907487931 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50390579837952 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50390685536256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11581485 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526178326027 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37882456256512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49280746201264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49280729161904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49862956630528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 527728905728 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496862 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1458900 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5084623 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527640246784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528425962496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528425962496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6506696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49881981507923 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527048607157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842384804814754e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092250 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6496541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528514628608 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 545024893952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚51 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4861980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 422.345643439 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123146240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49355019687296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50466515219968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49882085333813 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4862151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217255936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37394378333696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528514628608 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37394284224000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚649 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50466620925440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49938089257472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3401008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11588791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528425962496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528425962496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6506696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49881981507923 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527048607157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842384804814754e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092250 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6496541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528514628608 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 545024893952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚51 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4861980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 422.345643439 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123146240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49355019687296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50466515219968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49882085333813 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4862151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217255936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37394378333696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528514628608 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37394284224000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚649 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50466620925440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49938089257472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3401008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11588791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54187413093376 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6979573 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40145127005696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 584750370816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 566990187008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466073 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54187518791680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40729877376512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52995272470864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53560776114117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40145032896000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 584620925952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53560672288339 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466073 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚699 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 567078845952 [Kñf#ºDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5213884 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3648756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12445644 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53620439945728Û¾ÂB½'Ÿ·O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 567078845952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5213884 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3648756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12445644 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53620439945728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6989714 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53620422906368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52995255431504 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 565503643253 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6989714 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1565307 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53620422906368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54187413093376 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6979573 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40145127005696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 584750370816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 566990187008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466073 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54187518791680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40729877376512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52995272470864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53560776114117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40145032896000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 584620925952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53560672288339 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466073 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚699 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6999548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53695589612032 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6999548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40786822864896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5221059 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53069545956896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1567251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 566287137965 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54263348475392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53069562996256 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53695572572672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40201170624000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54263454180864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 567864568832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚699 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6989393 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3653808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53635746308331 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 566200351435 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5221230 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5473700 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 567864568832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 567775902720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3653812 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585558131200 _runtime‚ 453.781417685 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40201264733696 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 585432887808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53635850134221 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5473700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1567418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12473250 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6989393 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53695572572672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125243392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 143654912002á°¤Ç&¢ÁM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125243392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842385119172728e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6999548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53695589612032 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6999548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40786822864896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5221059 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53069545956896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1567251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 566287137965 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54263348475392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53069562996256 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53695572572672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40201170624000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54263454180864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 567864568832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚699 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6989393 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3653808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53635746308331 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 566200351435 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5221230 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5473700 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 567864568832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 567775902720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3653812 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585558131200 _runtime‚ 453.781417685 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40201264733696 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 585432887808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53635850134221 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5473700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1567418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12473250 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6989393 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚536955725726727\¾T†0£+ _wandbruntime‚453  _runtime‚453¢Ù`!n¤ø %loss/total_loss‚24.35730797263257 0loss/FRt1/assignment_loss‚3.9930175120337363 1loss/FBt1/assignment_loss‚0.21534610974118304 0loss/SRqqt1/detection_loss‚2.571714169280353 0loss/SRqqt2/detection_loss‚0.812869572043515 epoch‚1 % _timestamp‚1.7842385119182441e+09 _step‚56 .loss/FRt1/detection_loss‚0.446414940087637 /loss/FRt2/detection_loss‚1.2884647638941975 _runtime‚ 453.781882066 0loss/FRt2/assignment_loss‚2.5114348357450877 .loss/FBt2/detection_loss‚5.258640651397396 trainer/global_step‚699 2loss/SRqqt1/assignment_loss‚1.5414580218293654 2logëqìss/SRqqt2/assignment_loss‚1.2056286063647685 0loss/FBt2/assignment_loss‚0.2522210448503363 .loss/FBt1/detection_loss‚4.260097745364992 „¥ü 0loss/SRqqt2/detection_loss‚0.812869572043515 epoch‚1 % _timestamp‚1.7842385119182441e+09 _step‚56 .loss/FRt1/detection_loss‚0.446414940087637 /loss/FRt2/detection_loss‚1.2884647638941975 _runtime‚ 453.781882066 0loss/FRt2/assignment_loss‚2.5114348357450877 .loss/FBt2/detection_loss‚5.258640651397396 trainer/global_step‚699 2loss/SRqqt1/assignment_loss‚1.5414580218293654 2loss/SRqqt2/assignment_loss‚1.2056286063647685 0loss/FBt2/assignment_loss‚0.2522210448503363 .loss/FBt1/detection_loss‚4.260097745364992 %loss/total_loss‚24.35730797263257 0loss/FRt1/assignment_loss‚3.9930175120337363 1loss/FBt1/assignment_loss‚0.21534610974118304 0loss/SRqqt1/detection_loss‚2.5717141692803538He¦f¦‚(:Û »£åÒÕÉûA network.recv‚170742' disk.nvme1c1n1.out‚2382.13623046875' disk.nvme0c0n1.out‚2763.11181640625 proc.cpu.threads‚16 network.sent‚908624% disk./.usageGB‚1140.1659393310547 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 12.76953125 disk.dm-1.in‚0! disk.dm-1.out‚136.22900390625 disk.rbd0.out‚0) disk./.usagePercent‚74.26581594730857 disk.dm-0.in‚ 4.55859375 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125% disk.nvme0n1.out‚2763.11181640625$ memory_percent‚89.11395072937012( proc.memory.availableMB‚ 1783.5703125# proc.memory.rssMB‚ 2843.69140625% disk.nvme1n1.out‚2382.13623046875 disk.md0.in‚0" disk.dm-0.out‚4981.69677734375 disk.rbd0.in‚0* proc.memory.percent‚17.356514930725098 cpu‚25.681505267326504¤œ%%œ§‚(:‘ »£åÒæýëA gpu.0.gpu‚17.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.58966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áˆ1!]¨‚(:Ò Ê£åÒþÞB network.recv‚175207( disk./.usagePercent‚74.2658696176951 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2383.0556640625 disk.nvme1n1.in‚ 1.6640625# proc.memory.rssMB‚ 2843.69140625$ disk./.usageGB‚1140.166763305664& disk.nvme0c0n1.out‚2768.7900390625$ disk.nvme1n1.out‚2383.0556640625 disk.md0.out‚ 12.9453125! disk.dm-0.out‚4987.7607421875 disk.dm-1.in‚0 proc.cpu.threads‚16 network.sent‚971056 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125$ disk.nvme0n1.out‚2768.7900390625$ memory_percent‚89.11411762237549) proc.memory.availableMB‚ 1783.54296875 cpu‚25.67951534318118 disk.md0.in‚0 disk.dm-0.in‚ 4.55859375 disk.dm-1.out‚136.3544921875* proc.memory.percent‚17.356514930725098’li„©‚(:’ Ê£åÒ¸ÿA gpu.0.gpu‚90.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚224.411' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.80366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›„¢Å0ª+ _wandbruntime‚484  _runtime‚484=yÈ Ä«¾ _step‚57 & lr-AdamW/pg1‚0.0002364822323480858 & lr-AdamW/pg2‚0.0002364822323480858 trainer/global_step‚749 % _timestamp‚1.7842385425307066e+09 _runtime‚ 484.394678014z&ƒ´È¬Â & lr-AdamW/pg1‚0.0002364822323480858 & lr-AdamW/pg2‚0.0002364822323480858 trainer/global_step‚749 % _timestamp‚1.7842385425307066e+09 _runtime‚ 484.394678014 _step‚579á²¼0­+ _wandbruntime‚484  _runtime‚484GE1F¸'®²O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5847523 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57984352047104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 606340127232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1671468 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42951919296000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57377906221568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57314437088747 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 604742174061 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57377923260928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42952013405696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7472425 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3901552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5573020 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 606428786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 625260545536 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 625131100672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 606340127232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57984246348800 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 604655387643 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57984352047104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7482566 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5573196 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43577273951232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 484.39834557 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43577050396672 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5847521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57314540914525 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 606428786176 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7472425 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5847521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _step‚58 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57984246348800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 % _timestamp‚1.7842385425315845e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3901556 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57377923260928ÁîýE¼'¯¶O YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 606340127232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1671468 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42951919296000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57377906221568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57314437088747 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 604742174061 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57377923260928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42952013405696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7472425 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3901552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5573020 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 606428786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 625260545536 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 625131100672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 606340127232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57984246348800 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 604655387643 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57984352047104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7482566 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5573196 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43577273951232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 484.39834557 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43577050396672 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5847521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57314540914525 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 606428786176 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7472425 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5847521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _step‚58 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57984246348800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 % _timestamp‚1.7842385425315845e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3901556 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57377923260928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5847523 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57984352047104:îl_Ë0°+ _wandbruntime‚484  _runtime‚484“€ ±¼M ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5580116 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3906612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 626070951936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 605438882243 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7492400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 607125842944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225644544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58060287436288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58060181730816 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 607214509056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43633996441088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 607214509056 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5855150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57453072927232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚749 _runtime‚ 484.788980706 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 625939417088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5855152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784238542924838e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5855150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58060287436288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57453072927232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7482245 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5855152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 607125842944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43008057024000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131534848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130ðv0Æ&²ÀM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58060287436288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57453072927232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7482245 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5855152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 607125842944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43008057024000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131534848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5580116 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3906612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 626070951936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 605438882243 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7492400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 607125842944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225644544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58060287436288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58060181730816 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 607214509056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43633996441088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 607214509056 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5855150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57453072927232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚749 _runtime‚ 484.788980706 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 625939417088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5855152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784238542924838e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5855150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 F^¶‚(:Ó Ù£åÒ¥ä‚B disk.md0.out‚ 13.26953125% disk./.usageGB‚1140.2618675231934 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2383.8447265625! disk.dm-0.out‚5032.4677734375 proc.cpu.threads‚16 network.recv‚178872$ disk.nvme1n1.out‚2383.8447265625 disk.dm-1.out‚136.4443359375 disk.rbd0.out‚0' proc.memory.availableMB‚ 1783.671875* proc.memory.percent‚17.356514930725098) disk./.usagePercent‚74.27206432328242& disk.nvme0c0n1.out‚2813.5166015625$ disk.nvme0n1.out‚2813.5166015625 disk.nvme1n1.in‚ 1.6640625 disk.dm-0.in‚ 4.55859375 disk.rbd0.in‚0 network.sent‚975710 disk.nvme0n1.in‚ 2.89453125 disk.dm-1.in‚0$ memory_percent‚89.11333084106445# proc.memory.rssMB‚ 2843.69140625 cpu‚25.67717849488785 disk.md0.in‚0Lý½ž·‚(:“ Ù£åÒïôA gpu.0.gpu‚74.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚113.193' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.730999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Ä äc¸‚(:Ø è£åÒÜ›àA( proc.memory.availableMB‚ 1783.3515625* proc.memory.percent‚17.356514930725098 network.sent‚ 1.037683e+06) disk./.usagePercent‚74.27079685734842$ disk.nvme0n1.out‚2854.4462890625& disk.nvme1c1n1.out‚2384.6748046875$ disk.nvme1n1.out‚2384.6748046875 disk.dm-0.in‚ 4.55859375 disk.dm-1.in‚0 proc.cpu.threads‚16& disk.nvme0c0n1.out‚2854.4462890625 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 13.52734375 disk.dm-0.out‚5073.462890625 disk.dm-1.out‚ 136.63671875% disk./.usageGB‚1140.2424087524414 disk.nvme0n1.in‚ 2.89453125# proc.memory.rssMB‚ 2843.69140625 cpu‚25.675470876581777 network.recv‚182708 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.md0.in‚0$ memory_percent‚89.11528587341309yC$¹‚(:„ è£åÒ®éÏA gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚207.27' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚69.09 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T"ó0º+ _wandbruntime‚515  _runtime‚515ˆÿ>‚Å»¿ trainer/global_step‚799 $ _timestamp‚1.784238573936102e+09 _runtime‚ 515.800077505 _step‚61 ' lr-AdamW/pg1‚0.00023642541562012874 ' lr-AdamW/pg2‚0.00023642541562012874‚ÞQɼà ' lr-AdamW/pg2‚0.00023642541562012874 trainer/global_step‚799 $ _timestamp‚1.784238573936102e+09 _runtime‚ 515.800077505 _step‚61 ' lr-AdamW/pg1‚0.00023642541562012874=.60½+ _wandbruntime‚515  _runtime‚515ñv–Á¹'¾³O SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6228973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217263104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45758805696000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61781185302528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4154356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7975418 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 643980704869 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6228971 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14194248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46424436056064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 % _timestamp‚1.7842385739369826e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚62 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 645690067456 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5932652 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 643893918451 _runtime‚ 515.804392192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61781079604224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61135406576128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5932816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚164 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61068201889155 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60424325010064 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46424653319168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 665753513472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6228973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1778460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7965277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4154352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61068305714933 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6228971 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61135406576128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61781079604224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7965277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123153408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 645690067456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61781185302528!L>~ο·O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61135406576128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61781079604224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7965277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123153408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 645690067456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61781185302528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6228973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217263104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45758805696000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61781185302528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4154356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7975418 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 643980704869 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6228971 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14194248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46424436056064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 % _timestamp‚1.7842385739369826e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚62 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 645690067456 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5932652 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6ùd¶ï43893918451 _runtime‚ 515.804392192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61781079604224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61135406576128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5932816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚164 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61068201889155 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60424325010064 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46424653319168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 665753513472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6228973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1778460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7965277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4154352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61068305714933 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6228971 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130>Lë{0À+ _wandbruntime‚516  _runtime‚516ŸÌNÃ&Á½M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6236600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127340544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666436661248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60498615535456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4159412 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46481380085248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45815037533696 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 644764199581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4159408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61210556242432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61210539203072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 644677413051 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46481601535488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1780288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 646475783168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60498598496096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 646475783168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61857014986240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7975097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221450240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14211697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45814943424000 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 666564001792 _runtime‚ 516.196003113 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6236600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6236602 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5939869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5939696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7985252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚63 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61857120691712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 646564449280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ˜ðtÇ&ÂÁM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 646475783168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61857014986240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7975097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221450240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14211697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45814943424000 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 666564001792 _runtime‚ 516.196003113 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6236600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6236602 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5939869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5939696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7985252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚63 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61857120691712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 646564449280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6236600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127340544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666436661248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60498615535456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4159412 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46481380085248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45815037533696 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 644764199581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4159408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61210556242432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61210539203072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 644677413051 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46481601535488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1780288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 646475783168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60498598496096?ãöðº0Ã+ _wandbruntime‚516  _runtime‚516 µ”ÇÄû 0loss/FRt1/assignment_loss‚3.9348836635931943 /loss/FBt2/detection_loss‚2.2980700921775266 &loss/total_loss‚20.555939619136232 _runtime‚ 516.196488114 _step‚64 0loss/FRt2/assignment_loss‚2.2410875858181583 2loss/SRqqt1/assignment_loss‚1.6926734296333354 2loss/SRqqt2/assignment_loss‚1.1603281122985507 /loss/FRt1/detection_loss‚0.4956982930202801 .loss/FRt2/detection_loss‚1.306810522604156 0loss/SRqqt1/detection_loss‚2.079198111648244 1loss/SRqqt2/detection_loss‚0.7380412172548863 -loss/FBt1/detection_loss‚4.26790087509316 1loss/FBt1/assignment_loss‚0.14264928803281277 epoch‚1 trainer/global_step‚799 1loss/FBt2/assignment_loss‚0.19859842796192984 % _timestamp‚1.7842385743328483e+09Î JÅÿ 2loss/SRqqt2/assignment_loss‚1.1603281122985507 /loss/FRt1/detection_loss‚0.4956982930202801 .loss/FRt2/detection_loss‚1.306810522604156 0loss/SRqqt1/detection_loss‚2.079198111648244 1loss/SRqqt2/detection_loss‚0.7380412172548863 -loss/FBt1/detection_loss‚4.26790087509316 1loss/FBt1/assignment_loss‚0.14264928803281277 epoch‚1 trainer/global_step‚799 1loss/FBt2/assignment_loss‚0.19859842796192984 % _timestamp‚1.7842385743328483e+09 0loss/FRt1/assignment_loss‚3.9348836635931943 /loss/FBt2/detection_loss‚2.2980700921775266 &loss/total_loss‚20.555939619136232 _runtime‚ 516.196488114 _step‚64 0loss/FRt2/assignment_loss‚2.2410875858181583 2loss/SRqqt1/assignment_loss‚1.6926734296333354@=~×_Æ‚(:Ô ÷£åÒ÷„†B cpu‚25.726771092984613 proc.cpu.threads‚16 disk.dm-0.in‚ 4.55859375& disk.nvme0c0n1.out‚2900.3994140625& disk.nvme1c1n1.out‚2385.3857421875' proc.memory.availableMB‚ 1738.078125 disk.dm-0.out‚5119.376953125 disk.nvme0n1.in‚ 2.89453125$ disk.nvme0n1.out‚2900.3994140625$ disk.nvme1n1.out‚2385.3857421875# memory_percent‚89.3916130065918* proc.memory.percent‚17.631840705871582) disk./.usagePercent‚74.27030984828534 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0 disk.md0.out‚ 13.828125 network.sent‚ 1.043074e+06 network.recv‚186812% disk./.usageGB‚1140.2349319458008 disk.dm-1.in‚0 disk.dm-1.out‚ 136.72265625# proc.memory.rssMB‚ 2888.80078125¥)dÇ‚(:’ ÷£åÒÁعB gpu.0.gpu‚86.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚222.652' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.21733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°)õ«0È+ _wandbruntime‚529  _runtime‚529n¦×¨É’| ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.021688375622034073 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23814314603805542 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05697537213563919 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.189121812582016 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2447916567325592 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9899935126304626 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9926929473876953 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974044561386108 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49537035822868347 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2065824270248413 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.23566804826259613 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12569795548915863 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5443999767303467 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053923193365335464 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5213358998298645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9750794172286987 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9828858971595764 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9889086484909058 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.24649055302143097 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2135668247938156 *jet/accuracy_1_of_4‚0.6852413415908813 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841532111167908 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12774837017059326 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21630814671516418 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 particle/accuracy_4_of_5‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9595810770988464 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06496089696884155 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35847800970077515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.02265283092856407 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7129629254341125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9640355706214905 +jet/accuracy_3_of_3‚0.01816697046160698 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.060346223413944244 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27266788482666016 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.17592591047286987 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.35704541206359863 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32015520334243774 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04565818980336189 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_2_of_4‚0.24114231765270233 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.042231298983097076 /particle/accuracy_4_of_4‚0.8988019227981567 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23139651119709015 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.5685949921607971 +particle/sensitivity‚0.5234986543655396 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9557174444198608 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00915488414466381 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007016556803137064 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23371033370494843 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05548469349741936 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9757009148597717 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20596054196357727 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6157407760620117 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.231914684176445 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.7186252474784851 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚20.359759082321666 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.44505059719085693 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19878871738910675 /particle/accuracy_3_of_3‚0.9985920190811157 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5961538553237915 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4894760251045227 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.21363413333892822 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4667470157146454 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33702120184898376 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 *"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_4‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4615930914878845 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9306616187095642 7 Purityü÷y„p&/1FRt*SRqqt2FBt/FBt_purity‚0.9336357116699219 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.154931902885437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.1783333271741867 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.2294151484966278 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16975246369838715 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.21476824581623077 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.24966265261173248 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17581622302532196 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3484223186969757 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1856013983488083 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19110055267810822 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1525578796863556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05395293980836868 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.19088806211948395 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 'particle/f_score‚0.5458136796951294 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.4387764632701874 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3118192255496979 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2013888955116272 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1640625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7885167002677917 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1391267478466034 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2018720805644989 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9872979521751404 /particle/accuracy_3_of_4‚0.9979296326637268 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9978355169296265 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4821050763130188 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.47212013602256775 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02666666731238365 +particle/specificity‚0.8920553922653198 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9774070978164673 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021242162212729454 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0643220990896225 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34560999274253845 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3222600519657135 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9922166466712952 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.472335547208786 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.30179518461227417 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23371033370494843 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7000800371170044 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2353660762310028 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2817707061767578 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4981270730495453 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7481454014778137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9926929473876953 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13342326879501343 (particle/accuracy‚0.8128951787948608 trainer/global_step‚799 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009713279083371162 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9796943664550781 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023398001212626696 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23115669190883636 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.35812368988990784 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 $ _timestamp‚1.784238587560701e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23412609100341797 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009562172926962376 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9777733683586121 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6validation_average_jet_accuracy‚0.1904042512178421 _runtime‚ 529.425372068 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.591940701007843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9612433314323425 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.056128133088350296 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7853595018386841 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚65 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08576266467571259 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7129629254341125 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23220223188400269 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.056128133088350296 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20596054196357727 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.1412319839000702 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.950554370880127 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04856362193822861 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.23155932128429413 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13677780330181122 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9011086225509644 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.23566804826259613 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004382017999887466 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1678529530763626 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23744846880435944 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9337105751037598 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.4212559163570404 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22314874827861786 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21976232528686523 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9817206859588623 *"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9939354658126831 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46587300300598145 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8481082916259766 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.42442071437835693 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17347650229930878 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3279598355293274 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23811781406402588 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003800220787525177 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 jet/accuracy_4_of_5‚0 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0071800025179982185 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2836177945137024 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5257601141929626 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_2‚0.05892710015177727°¨ÿv>Ê–| ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1525578796863556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05395293980836868 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.19088806211948395 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 'particle/f_score‚0.5458136796951294 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.4387764632701874 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3118192255496979 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2013888955116272 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1640625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7885167002677917 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1391267478466034 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2018720805644989 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9872979521751404 /particle/accuracy_3_of_4‚0.9979296326637268 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9978355169296265 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4821050763130188 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.47212013602256775 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02666666731238365 +particle/specificity‚0.8920553922653198 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9774070978164673 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021242162212729454 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0643220990896225 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34560999274253845 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3222600519657135 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9922166466712952 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.472335547208786 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.30179518461227417 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23371033370494843 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7000800371170044 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2353660762310028 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2817707061767578 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4981270730495453 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7481454014778137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9926929473876953 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13342326879501343 (particle/accuracy‚0.8128951787948608 trainer/global_step‚799 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009713279083371162 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9796943664550781 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023398001212626696 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23115669190883636 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.35812368988990784 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 $ _timestamp‚1.784238587560701e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23412609100341797 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009562172926962376 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9777733683586121 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6validation_average_jet_accuracy‚0.1904042512178421 _runtime‚ 529.425372068 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.591940701007843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9612433314323425 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.056128133088350296 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7853595018386841 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚65 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08576266467571259 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7129629254341125 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23220223188400269 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.056128133088350296 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20596054196357727 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.1412319839000702 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.950554370880127 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04856362193822861 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.23155932128429413 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13677780330181122 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9011086225509644 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.23566804826259613 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004382017999887466 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1678529530763626 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23744846880435944 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9337105751037598 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.4212559163570404 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22314874827861786 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21976232528686523 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9817206859588623 *"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9939354658126831 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46587300300598145 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8481082916259766 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.42442071437835693 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17347650229930878 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3279598355293274 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23811781406402588 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003800220787525177 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 jet/accuracy_4_of_5‚0 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0071800025179982185 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2836177945137024 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5257601141929626 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_2‚0.05892710015177727 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.021688375622034073 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23814314603805542 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05697537213563919 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.189121812582016 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2447916567325592 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9899935126304626 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9926929473876953 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974044561386108 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49537035822868347 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2065824270248413 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.23566804826259613 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12569795548915863 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5443999767303467 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053923193365335464 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5213358998298645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9750794172286987 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9828858971595764 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9889086484909058 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.24649055302143097 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2135668247938156 *jet/accuracy_1_of_4‚0.6852413415908813 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841532111167908 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12774837017059326 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21630814671516418 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 particle/accuracy_4_of_5‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9595810770988464 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06496089696884155 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35847800970077515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.02265283092856407 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7129629254341125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9640355706214905 +jet/accuracy_3_of_3‚0.01816697046160698 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.060346223413944244 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27266788482666016 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.17592591047286987 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.35704541206359863 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32015520334243774 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04565818980336189 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_2_of_4‚0.24114231765270233 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.042231298983097076 /particle/accuracy_4_of_4‚0.8988019227981567 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23139651119709015 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.5685949921607971 +particle/sensitivity‚0.5234986543655396 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9557174444198608 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00915488414466381 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007016556803137064 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23371033370494843 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05548469349741936 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9757009148597717 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20596054196357727 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6157407760620117 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.231914684176445 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.7186252474784851 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚20.359759082321666 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.44505059719085693 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19878871738910675 /particle/accuracy_3_of_3‚0.9985920190811157 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5961538553237915 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4894760251045227 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.21363413333892822 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4667470157146454 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33702120184898376 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 *"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_4‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4615930914878845 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9306616187095642 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9336357116699219 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.154931902885437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.1783333271741867 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.2294151484966278 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16975246369838715 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.21476824581623077 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.24966265261173248 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17581622302532196 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3484223186969757 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1856013983488083 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19110055267810822 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925A3Y…gË‚(:Ü †¤åÒüÄ’B disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme1n1.in‚ 1.6640625$ disk.nvme1n1.out‚2514.1279296875" disk.dm-0.out‚5223.11962890625# memory_percent‚89.7777795791626) proc.memory.availableMB‚ 1674.80859375) disk./.usagePercent‚74.26988595162128% disk./.usageGB‚1140.2284240722656' disk.nvme0c0n1.out‚3004.28466796875 disk.nvme0n1.in‚ 2.89453125% disk.nvme0n1.out‚3004.28466796875 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2514.1279296875 disk.md0.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 4.55859375" proc.memory.rssMB‚ 2887.8828125* proc.memory.percent‚17.626237869262695 proc.cpu.threads‚16 network.sent‚ 1.105289e+06 disk.md0.out‚ 14.33984375 disk.dm-1.in‚0 disk.dm-1.out‚ 264.501953125 cpu‚25.748638033046028 network.recv‚192558¶ñ`ÚÌ‚(:… †¤åÒÇ«ÅB gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.074' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈÒ#Ð`Í‚(:Õ •¤åÒ˜ë„B network.recv‚197368 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3036.35595703125 disk.nvme1c1n1.in‚ 1.6640625) proc.memory.availableMB‚ 1674.42578125" proc.memory.rssMB‚ 2887.8828125 cpu‚25.745602880012083 proc.cpu.threads‚16 network.sent‚ 1.161482e+06( disk./.usagePercent‚74.0988843974064% disk./.usageGB‚1137.6031227111816" disk.dm-0.out‚5254.93212890625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 2.89453125# disk.nvme1c1n1.out‚ 2514.8828125* proc.memory.percent‚17.626237869262695 disk.dm-0.in‚ 4.55859375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 2.89453125! disk.nvme1n1.out‚ 2514.8828125 disk.md0.out‚ 14.77734375# memory_percent‚89.7801160812378 disk.dm-1.out‚ 264.552734375% disk.nvme0n1.out‚3036.35595703125 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0Íæƒn›Î‚(: •¤åÒŠž¹B gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚191.162' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.72066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îõã0Ï+ _wandbruntime‚560  _runtime‚560Ú@<|Åп _step‚66 ' lr-AdamW/pg1‚0.00023636101407973755 ' lr-AdamW/pg2‚0.00023636101407973755 trainer/global_step‚849 % _timestamp‚1.7842386191175969e+09 _runtime‚ 560.98154434ÀQ̤ÉÑà ' lr-AdamW/pg1‚0.00023636101407973755 ' lr-AdamW/pg2‚0.00023636101407973755 trainer/global_step‚849 % _timestamp‚1.7842386191175969e+09 _runtime‚ 560.98154434 _step‚66BáÊ(0Ò+ _wandbruntime‚560  _runtime‚560÷›ËQGÓ³O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65112318315856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚67 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6447411 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65112301276496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49369151410688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732223 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65810952810095 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50090854102016 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8540631 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 % _timestamp‚1.7842386191184638e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66589935228928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65889360147968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 700469382144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50091069267456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6732221 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1932004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65889377187328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66589829530112 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66589829530112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 700469382144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8540631 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8550773 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65889377187328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4515411 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66589829530112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 700469382144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8540631 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8550773 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65889377187328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4515411 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65112301276496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49369151410688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732223 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65810952810095 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50090854102016 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8540631 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 % _timestamp‚1.7842386191184638e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66589935228928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65889360147968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584CULíM0Õ+ _wandbruntime‚561  _runtime‚561l%ßÛÁ&Ö»M _runtime‚ 561.37634455 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4520463 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6454620 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65186608841248 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8550451 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49425289138688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6454794 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4520467 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 701343764480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65885923004305 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6739852 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66665870618112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49425195028992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 trainer/global_step‚849 _step‚68 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8560607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65886026830199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6739850 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 722608566272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65964526853632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6454794 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4520467 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 701343764480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65885923004305 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6739852 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66665870618112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49425195028992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 trainer/global_step‚849 _step‚68 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8560607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65886026830199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6739850 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 722608566272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65964526853632 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6454620 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65186608841248 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8550451DO;+§0Ø+ _wandbruntime‚561  _runtime‚561Î$x£Ùû epoch‚2 0loss/FRt1/assignment_loss‚3.8338696806858747 /loss/FRt2/detection_loss‚1.2814997593740751 0loss/SRqqt1/detection_loss‚2.381827054093087 /loss/FBt2/detection_loss‚3.1557532143558005 &loss/total_loss‚19.466163717417157 2loss/SRqqt2/assignment_loss‚1.1283247200586126 trainer/global_step‚849 _runtime‚ 561.37677697 _step‚69 0loss/FRt2/assignment_loss‚2.2802562407452425 1loss/FBt1/assignment_loss‚0.09920504862078977 /loss/FBt1/detection_loss‚2.3289717936678547 1loss/SRqqt1/assignment_loss‚1.462905920596886 0loss/FBt2/assignment_loss‚0.2631258927694511 /loss/FRt1/detection_loss‚0.5126867823199763 1loss/SRqqt2/detection_loss‚0.7377376101295069 % _timestamp‚1.7842386195131557e+09°=ñ|Úÿ 0loss/FRt2/assignment_loss‚2.2802562407452425 1loss/FBt1/assignment_loss‚0.09920504862078977 /loss/FBt1/detection_loss‚2.3289717936678547 1loss/SRqqt1/assignment_loss‚1.462905920596886 0loss/FBt2/assignment_loss‚0.2631258927694511 /loss/FRt1/detection_loss‚0.5126867823199763 1loss/SRqqt2/detection_loss‚0.7377376101295069 % _timestamp‚1.7842386195131557e+09 epoch‚2 0loss/FRt1/assignment_loss‚3.8338696806858747 /loss/FRt2/detection_loss‚1.2814997593740751 0loss/SRqqt1/detection_loss‚2.381827054093087 /loss/FBt2/detection_loss‚3.1557532143558005 &loss/total_loss‚19.466163717417157 2loss/SRqqt2/assignment_loss‚1.1283247200586126 trainer/global_step‚849 _runtime‚ 561.37677697 _step‚69E”|J£bÛ‚(:× ¤¤åÒæ†éA) disk./.usagePercent‚74.09899496834163 disk.nvme0n1.in‚ 2.89453125! disk.nvme0n1.out‚ 3060.8515625 disk.dm-1.in‚0$ memory_percent‚89.78044986724854) proc.memory.availableMB‚ 1674.37109375# proc.memory.rssMB‚ 2887.88671875 network.recv‚200840! disk.dm-1.out‚264.74755859375 disk.rbd0.out‚0* proc.memory.percent‚17.626261711120605 cpu‚25.741877019962402' disk.nvme1c1n1.out‚2515.85888671875 disk.md0.in‚0 disk.md0.out‚ 15.20703125 disk.rbd0.in‚0 proc.cpu.threads‚16% disk./.usageGB‚1137.6048202514648 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625% disk.nvme1n1.out‚2515.85888671875 disk.dm-0.in‚ 4.55859375 disk.dm-0.out‚5279.294921875 disk.nvme0c0n1.in‚ 2.89453125# disk.nvme0c0n1.out‚ 3060.8515625 network.sent‚ 1.16767e+06åÖ®ÑÜ‚(:’ ¤¤åÒäòÜA gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.112' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.03733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ˆÈcÝ‚(:Ø ³¤åÒ·ùÞA network.sent‚ 1.230648e+06 disk.dm-0.in‚ 4.55859375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625% disk.nvme1n1.out‚2516.69873046875 proc.cpu.threads‚16 network.recv‚205631 disk.dm-0.out‚5332.470703125$ disk.nvme0c0n1.out‚ 3114.26953125' disk.nvme1c1n1.out‚2516.69873046875# proc.memory.rssMB‚ 2887.88671875 cpu‚25.738325663432374( disk./.usagePercent‚74.0969669234576! disk.dm-1.out‚264.78271484375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 2.89453125" disk.nvme0n1.out‚ 3114.26953125 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0) proc.memory.availableMB‚ 1674.55078125% disk./.usageGB‚1137.5736846923828 disk.dm-1.in‚0 disk.md0.out‚ 15.6953125$ memory_percent‚89.77935314178467* proc.memory.percent‚17.626261711120605+˜œ¿œÞ‚(:‘ ³¤åÒõ‘B gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.377' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.458999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ȶm£0ß+ _wandbruntime‚592  _runtime‚592œÝBÆàÀ _step‚70 ' lr-AdamW/pg1‚0.00023628903186330286 ' lr-AdamW/pg2‚0.00023628903186330286 trainer/global_step‚899 % _timestamp‚1.7842386505106401e+09 _runtime‚ 592.374590558Ÿ™fÊáÄ ' lr-AdamW/pg2‚0.00023628903186330286 trainer/global_step‚899 % _timestamp‚1.7842386505106401e+09 _runtime‚ 592.374590558 _step‚70 ' lr-AdamW/pg1‚0.00023628903186330286Fí:Å*0â+ _wandbruntime‚592  _runtime‚592!ßÆ’¹'ã³O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113671 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9033483 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 737873025047 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4768207 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68826844585456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9033483 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809295 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70386768484352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52175943700992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52176037810688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68826827546096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386662785536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16157298 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2041259 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52938620824576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69564717610503 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69646860502528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386662785536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 739819322368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682²Îw½'ä·O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70386768484352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52175943700992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52176037810688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68826827546096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386662785536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16157298 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2041259 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52938620824576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69564717610503 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69646860502528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386662785536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 739819322368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113671 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9033483 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 737873025047 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4768207 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68826844585456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9033483 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809295 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142G— Ûï0å+ _wandbruntime‚592  _runtime‚592dg畾æ½M REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70462703873536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114757120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121302 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816714 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 738569733225 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚72 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69639791630607 % _timestamp‚1.7842386509056807e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16164603 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70462598167552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2043447 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69639687804713 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.alZ‰Ílocated‚ 740693704704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70462703873536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69721993129472 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 740605038080 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 763386186752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16164603 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4773267 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69721993129472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52995561725440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69722010168832 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9043303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 740693704704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2043273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52995352858624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9053459 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9043303 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚ 592.769840069 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52232081428992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68901118071488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552Ôë‡ÎÇ&çÁM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16174761 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69722010168832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52995561725440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69722010168832 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9043303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 740693704704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2043273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52995352858624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9053459 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9043303 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚ 592.769840069 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52232081428992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68901118071488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70462703873536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114757120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121302 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816714 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 738569733225 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚72 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69639791630607 % _timestamp‚1.7842386509056807e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16164603 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70462598167552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2043447 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69639687804713 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 740693704704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70462703873536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69721993129472 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 740605038080 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 763386186752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16164603 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4773267 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69721993129472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74183496040960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54982830100992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17021456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72541353815696 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495123 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495121 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777111555855 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74183496040960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73404326778368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7170024 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777024769433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54982924210688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5021007 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74183601739776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7169837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73318378585129 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73318482410911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9536477 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55785889840640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55786109200384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 803059739648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74183601739776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 _step‚75 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚73404343817728 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495123 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74183496040960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54982830100992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17021456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72541353815696 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495123 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495121 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777111555855 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceSV#víÁtatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74183496040960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73404326778368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7170024 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777024769433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54982924210688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5021007 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74183601739776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7169837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73318378585129 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73318482410911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9536477 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55785889840640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55786109200384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 803059739648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74183601739776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 _step‚75 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚73404343817728 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495123 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777895050567 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚76 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7502750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7177004 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 803995502592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73479493484032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5026063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7502752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7177194 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55038967828992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73393452605121 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 777808264033 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779954978304 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7502750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55843057441280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73479493484032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2150941 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73479476444672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73479476444672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74259431422976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72615661380448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72615644341088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 _runtime‚ 624.167176917 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55039061938688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17038905 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74259537128960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1;!É/Æ&÷ÀM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17038905 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777895050567 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚76 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7502750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7177004 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 803995502592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73479493484032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5026063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7502752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7177194 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55038967828992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73393452605121 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 777808264033 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779954978304 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7502750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55843057441280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73479493484032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2150941 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73479476444672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73479476444672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74259431422976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72615661380448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72615644341088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 _runtime‚ 624.167176917 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55039061938688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17038905 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74259537128960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960L·Yø0ø+ _wandbruntime‚624  _runtime‚6243 ¢Ëùü 0loss/SRqqt1/detection_loss‚2.328232820370941 /loss/FBt1/detection_loss‚1.9355129242551372 &loss/total_loss‚16.980271320516003 /loss/FBt2/detection_loss‚1.7407271193143998 epoch‚2 0loss/FRt1/assignment_loss‚3.6748083188449066 0loss/FRt2/assignment_loss‚1.8130159313442142 1loss/FBt1/assignment_loss‚0.11624343105344442 .loss/FRt1/detection_loss‚0.613483681482293 /loss/FRt2/detection_loss‚1.3418373464766284 trainer/global_step‚949 1loss/FBt2/assignment_loss‚0.16882461421864356 % _timestamp‚1.7842386823039882e+09 _step‚77 2loss/SRqqt1/assignment_loss‚1.5072169427780413 1loss/SRqqt2/assignment_loss‚1.070145902356242 1loss/SRqqt2/detection_loss‚0.6702222880211142 _runtime‚ 624.167639288êýÍ6ú€ 1loss/FBt2/assignment_loss‚0.16882461421864356 % _timestamp‚1.7842386823039882e+09 _step‚77 2loss/SRqqt1/assignment_loss‚1.5072169427780413 1loss/SRqqt2/assignment_loss‚1.070145902356242 1loss/SRqqt2/detection_loss‚0.6702222880211142 _runtime‚ 624.167639288 0loss/SRqqt1/detection_loss‚2.328232820370941 /loss/FBt1/detection_loss‚1.9355129242551372 &loss/total_loss‚16.980271320516003 /loss/FBt2/detection_loss‚1.7407271193143998 epoch‚2 0loss/FRt1/assignment_loss‚3.6748083188449066 0loss/FRt2/assignment_loss‚1.8130159313442142 1loss/FBt1/assignment_loss‚0.11624343105344442 .loss/FRt1/detection_loss‚0.613483681482293 /loss/FRt2/detection_loss‚1.3418373464766284 trainer/global_step‚949MCÊB’û‚(:‡ à¤åÒô¾îA gpu.0.gpu‚57.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚113.388' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¢îŸXü‚(:Í à¤åÒ¹¸‹B disk.rbd0.out‚0 disk.nvme0n1.in‚ 2.89453125 disk.nvme1n1.in‚ 1.6640625! disk.nvme1n1.out‚ 2519.5390625 disk.dm-0.in‚ 4.55859375 network.sent‚ 1.30395e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 265.08984375 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625* proc.memory.percent‚17.626261711120605 proc.cpu.threads‚16) disk./.usagePercent‚74.10311615830076# disk.nvme0n1.out‚3314.962890625# disk.nvme1c1n1.out‚ 2519.5390625 disk.md0.in‚0 disk.md0.out‚ 16.66796875 disk.dm-0.out‚5533.583984375$ memory_percent‚90.17515182495117' proc.memory.availableMB‚ 1609.703125% disk.nvme0c0n1.out‚3314.962890625# proc.memory.rssMB‚ 2887.88671875 cpu‚25.72939455670138 network.recv‚218696% disk./.usageGB‚1137.6680908203125 disk.rbd0.in‚0Ý ý‚(:’ ï¤åÒó„ÔA gpu.0.gpu‚57.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.159' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.71966666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ÏDªUþ‚(:Ê ï¤åÒ¼ÿðA$ disk.nvme0c0n1.out‚ 5029.33984375% disk./.usageGB‚1137.7125511169434 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 16.96484375 disk.dm-0.out‚ 7247.80078125 proc.cpu.threads‚16 disk.nvme0c0n1.in‚ 2.89453125" disk.nvme0n1.out‚ 5029.33984375 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0 disk.dm-0.in‚ 4.55859375) proc.memory.availableMB‚ 1609.12109375* proc.memory.percent‚17.626261711120605 disk.nvme0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625 network.sent‚ 1.364597e+06 network.recv‚223139( disk./.usagePercent‚74.1060121229079 disk.dm-1.out‚ 265.15625# disk.nvme1c1n1.out‚ 2520.1015625! disk.nvme1n1.out‚ 2520.1015625$ memory_percent‚90.17870426177979# proc.memory.rssMB‚ 2887.88671875 cpu‚25.726461596118305™B“0ÿ+ _wandbruntime‚654  _runtime‚6546«A&À½ & lr-AdamW/pg1‚0.0002361223443820317 & lr-AdamW/pg2‚0.0002361223443820317 trainer/global_step‚999 $ _timestamp‚1.784238712880409e+09 _runtime‚ 654.746412877 _step‚78;‡0·ÇÁ & lr-AdamW/pg1‚0.0002361223443820317 & lr-AdamW/pg2‚0.0002361223443820317 trainer/global_step‚999 $ _timestamp‚1.784238712880409e+09 _runtime‚ 654.746412877 _step‚78N{·µá0‚+ _wandbruntime‚654  _runtime‚654X’)\ƒ·O \IDeviceStatsMoniÄ ïw¤'tor.on_train_batch_start/active_bytes.large_pool.allocated‚77161827132928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 818519202816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5273811 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 818607862272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10029329 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77980329296384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77161810093568 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7876573 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843742692864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7876571 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17895758 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58633553303552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76255880085296 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77980329296384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 654.746508897 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5273807 _step‚79 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77161810093568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10029329 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7530570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2256557 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77072143385537 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58633336040960 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816263300241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚206 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2256759 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77161827132928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10019187 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7530364 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 843619539968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77072247211319 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76255897124656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 818519202816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 818607862272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚77980434995200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1´`|Á'„»O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58633336040960 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816263300241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚206 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2256759 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77161827132928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10019187 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7530364 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 843619539968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77072247211319 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76255897124656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 818519202816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 818607862272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚77980434995200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77161827132928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 818519202816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5273811 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 818607862272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10029329 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77980329296384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77161810093568 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7876573 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843742692864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7876571 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17895758 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58633553303552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76255880085296 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77980329296384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 654.746508897 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5273807 _step‚79 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77161810093568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10029329 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7530570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2256557 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77072143385537 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0OÏ1N„0…+ _wandbruntime‚655  _runtime‚655CCÇ&†ÁM O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817046794841 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77236959759872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚194 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77236959759872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884202 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77147321231423 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 % _timestamp‚1.7842387132752368e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78056370384384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7537534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57845948338688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5278863 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2258671 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884202 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77236976799232 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 819393585152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10039163 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77147217405529 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7537728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5278867 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78056370384384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77236976799232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 819304918528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76330170610688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10039163 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58690504034816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78056264678400 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 819304918528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77236959759872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚194 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77236959759872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884202 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77147321231423 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 % _timestamp‚1.7842387132752368e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78056370384384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7537534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57845948338688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5278863 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2258671 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884202 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77236976799232 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 819393585152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10039163 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77147217405529 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7537728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5278867 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78056370384384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77236976799232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 819304918528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76330170610688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10039163 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58690504034816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78056264678400 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 819304918528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817046794841 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0PÕFˆn0ˆ+ _wandbruntime‚655  _runtime‚655‰ˆ 9‰þ 1loss/FBt1/assignment_loss‚0.12312804834196707 /loss/FRt1/detection_loss‚0.5675019665620802 1loss/SRqqt2/detection_loss‚0.7817905559512388 &loss/total_loss‚20.113385218549453 _runtime‚ 655.139919349 _step‚81 1loss/FBt2/assignment_loss‚0.16892852180961365 /loss/FRt2/detection_loss‚1.4037339298561908 epoch‚2 trainer/global_step‚999 0loss/FRt1/assignment_loss‚3.4853733795172883 0loss/FRt2/assignment_loss‚1.8454223874078435 2loss/SRqqt2/assignment_loss‚1.0394069132063675 1loss/SRqqt1/detection_loss‚2.2874992711216824 % _timestamp‚1.7842387132762282e+09 2loss/SRqqt1/assignment_loss‚1.6257701784758487 /loss/FBt1/detection_loss‚1.1587523603050827 .loss/FBt2/detection_loss‚5.626077705994248$8wŠ‚ 1loss/FBt2/assignment_loss‚0.16892852180961365 /loss/FRt2/detection_loss‚1.4037339298561908 epoch‚2 trainer/global_step‚999 0loss/FRt1/assignment_loss‚3.4853733795172883 0loss/FRt2/assignment_loss‚1.8454223874078435 2loss/SRqqt2/assignment_loss‚1.0394069132063675 1loss/SRqqt1/detection_loss‚2.2874992711216824 % _timestamp‚1.7842387132762282e+09 2loss/SRqqt1/assignment_loss‚1.6257701784758487 /loss/FBt1/detection_loss‚1.1587523603050827 .loss/FBt2/detection_loss‚5.626077705994248 1loss/FBt1/assignment_loss‚0.12312804834196707 /loss/FRt1/detection_loss‚0.5675019665620802 1loss/SRqqt2/detection_loss‚0.7817905559512388 &loss/total_loss‚20.113385218549453 _runtime‚ 655.139919349 _step‚81Q_W‹‚(:… þ¤åÒ‹äôA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.128' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßlp–`Œ‚(:Õ þ¤åÒ¦ˆB% disk.nvme0n1.out‚5075.36376953125 disk.nvme1c1n1.in‚ 1.6640625# disk.nvme1n1.out‚2521.115234375! disk.dm-1.out‚265.31298828125 cpu‚25.72400010890605 network.sent‚ 1.371299e+06' disk./.usagePercent‚74.104008180003 disk.md0.out‚17.3125 proc.cpu.threads‚16 network.recv‚226948$ disk./.usageGB‚1137.681785583496 disk.nvme0c0n1.in‚ 2.89453125% disk.nvme1c1n1.out‚2521.115234375 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1609.25390625 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5075.36376953125 disk.dm-0.in‚ 4.55859375! disk.dm-0.out‚7293.9287109375$ memory_percent‚90.17789363861084# proc.memory.rssMB‚ 2887.88671875* proc.memory.percent‚17.626261711120605 disk.nvme0n1.in‚ 2.89453125FE‰‚(:’ ¥åÒȈçA gpu.0.gpu‚71.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.321' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.77366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û?eŽ‚(:Ú ¥åÒ†•ùA disk.md0.out‚ 17.703125 disk.dm-0.in‚ 4.55859375! disk.dm-0.out‚8540.3037109375 disk.rbd0.in‚0% disk.nvme0c0n1.out‚6105.443359375) disk./.usagePercent‚74.09707774286684% disk.nvme1n1.out‚2738.31591796875 disk.md0.in‚0 disk.nvme0c0n1.in‚ 2.89453125# disk.nvme0n1.out‚6105.443359375# memory_percent‚90.1768445968628) proc.memory.availableMB‚ 1609.42578125# proc.memory.rssMB‚ 2887.88671875 network.sent‚ 1.433552e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 2.89453125* proc.memory.percent‚17.626261711120605 proc.cpu.threads‚16% disk./.usageGB‚1137.5753860473633 disk.nvme1c1n1.in‚ 1.6640625! disk.dm-1.out‚265.36376953125 cpu‚25.721306017249766 network.recv‚231527' disk.nvme1c1n1.out‚2738.31591796875 disk.nvme1n1.in‚ 1.6640625\ß6’0+ _wandbruntime‚686  _runtime‚686O}uwÇÁ ' lr-AdamW/pg1‚0.00023602764982322063 ' lr-AdamW/pg2‚0.00023602764982322063 trainer/global_step‚1049 % _timestamp‚1.7842387442436888e+09 _runtime‚ 686.107641973 _step‚82½ÿÄ@Ë‘Å ' lr-AdamW/pg1‚0.00023602764982322063 ' lr-AdamW/pg2‚0.00023602764982322063 trainer/global_step‚1049 % _timestamp‚1.7842387442436888e+09 _runtime‚ 686.107641973 _step‚82RSÕY0’+ _wandbruntime‚686  _runtime‚686}3\½'“·O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81777162551808 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 857957802496 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 884160072192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80919310448128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 857869143040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 857869143040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10512039 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10522181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10512039 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258023 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81777162551808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80919310448128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7891183 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5526611 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2364401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 trainer/global_step‚1049 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79970423394256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 $ _timestamp‚1.784238744244565e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258021 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80919293408768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79970423394256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 $ _timestamp‚1.784238744244565e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258021 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80919293408768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81777162551808 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 857957802496 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 884160072192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80919310448128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 857869143040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 857869143040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10512039 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10522181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10512039 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258023 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81777162551808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80919310448128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7891183 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5526611 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2364401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 trainer/global_step‚1049 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.s°Q¢P&large_pool.peak‚682 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8265650 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10521859 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81853203639808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _runtime‚ 686.504871924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18787509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8265652 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60652834738688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80044713919648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 858654858752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80994460114432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60652740628992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 885089674752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80900982205937 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80994443075072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80994460114432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 858743525376 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8265652 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10532015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61537705060864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 884964431872 % _timestamp‚1.7842387446407464e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7898198 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5531667 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2366535 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 858654858752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8265650 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7898387 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81853203639808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80994443075072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0î•YÆÌ&—ÆM J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7898198 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5531667 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2366535 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 858654858752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8265650 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7898387 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81853203639808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80994443075072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8265650 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10521859 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81853203639808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _runtime‚ 686.504871924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18787509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8265652 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60652834738688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80044713919648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 858654858752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80994460114432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60652740628992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 885089674752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80900982205937 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80994443075072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80994460114432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 858743525376 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8265652 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10532015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61537705060864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 884964431872 % _timestamp‚1.7842387446407464e+09T3@ v0˜+ _wandbruntime‚686  _runtime‚686ŸÎi™û 2loss/SRqqt2/assignment_loss‚1.0608034751841944 /loss/FRt1/detection_loss‚0.5701953062252825 1loss/SRqqt1/detection_loss‚1.9506071837899346 1loss/SRqqt2/detection_loss‚0.7762104432461275 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.1330634125838163 .loss/FBt2/detection_loss‚2.722700670913414 % _timestamp‚1.7842387446417136e+09 0loss/FRt2/assignment_loss‚1.8513024299343428 2loss/SRqqt1/assignment_loss‚1.1760076140929814 .loss/FRt2/detection_loss‚1.444401579372289 epoch‚2 _step‚85 /loss/FRt1/assignment_loss‚3.380564925650221 1loss/FBt2/assignment_loss‚0.28364090377443874 /loss/FBt1/detection_loss‚1.7830917327459082 %loss/total_loss‚17.13258967751295 _runtime‚ 686.505315555Û•Àšÿ %loss/total_loss‚17.13258967751295 _runtime‚ 686.505315555 2loss/SRqqt2/assignment_loss‚1.0608034751841944 /loss/FRt1/detection_loss‚0.5701953062252825 1loss/SRqqt1/detection_loss‚1.9506071837899346 1loss/SRqqt2/detection_loss‚0.7762104432461275 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.1330634125838163 .loss/FBt2/detection_loss‚2.722700670913414 % _timestamp‚1.7842387446417136e+09 0loss/FRt2/assignment_loss‚1.8513024299343428 2loss/SRqqt1/assignment_loss‚1.1760076140929814 .loss/FRt2/detection_loss‚1.444401579372289 epoch‚2 _step‚85 /loss/FRt1/assignment_loss‚3.380564925650221 1loss/FBt2/assignment_loss‚0.28364090377443874 /loss/FBt1/detection_loss‚1.7830917327459082U² uü›‚(:’ œ¥åÒÁ™ÜA gpu.0.gpu‚68.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚210.701' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.23366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}+ç'iœ‚(:Þ œ¥åÒê”ðA disk.md0.out‚ 18.00390625 network.sent‚ 1.438463e+06) disk./.usagePercent‚74.03115559975345 disk.dm-0.in‚ 4.55859375" disk.dm-0.out‚8563.26025390625 disk.dm-1.in‚0# memory_percent‚90.1775598526001 cpu‚25.718723330176275 network.recv‚234840% disk./.usageGB‚1136.5633163452148 disk.rbd0.out‚0 disk.nvme0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.in‚0 proc.cpu.threads‚16 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6127.27001953125% disk.nvme0n1.out‚6127.27001953125 disk.dm-1.out‚265.4501953125) proc.memory.availableMB‚ 1609.30859375# proc.memory.rssMB‚ 2887.88671875* proc.memory.percent‚17.626261711120605 disk.nvme0c0n1.in‚ 2.89453125& disk.nvme1c1n1.out‚2740.1865234375 disk.nvme1n1.in‚ 1.6640625$ disk.nvme1n1.out‚2740.1865234375~ú\›‚(: «¥åÒ™Í÷A gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚112.474' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.49133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èôscž‚(:Ø «¥åÒϬ†B disk.md0.out‚ 18.4609375) proc.memory.availableMB‚ 1609.70703125 network.sent‚ 1.500739e+06) disk./.usagePercent‚74.03118467121283 disk.dm-0.in‚ 4.55859375 disk.nvme1n1.in‚ 1.6640625$ memory_percent‚90.17512798309326 proc.cpu.threads‚16 network.recv‚239215" disk.dm-0.out‚8569.37646484375 disk.dm-1.in‚0 disk.dm-1.out‚265.5322265625 disk.nvme0c0n1.in‚ 2.89453125' disk.nvme0c0n1.out‚6133.60107421875 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2741.0224609375$ disk./.usageGB‚1136.563762664795 disk.nvme0n1.in‚ 2.89453125% disk.nvme0n1.out‚6133.60107421875 disk.md0.in‚0 proc.memory.rssMB‚ 2887.46875) proc.memory.percent‚17.62371063232422 cpu‚25.71623009694758 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚2741.0224609375“y¤0Ÿ+ _wandbruntime‚717  _runtime‚717 KKåÅ ¿ _step‚86 ' lr-AdamW/pg1‚0.00023592539599972935 ' lr-AdamW/pg2‚0.00023592539599972935 trainer/global_step‚1099 % _timestamp‚1.7842387756537297e+09 _runtime‚ 717.5176919V’bÓÉ¡Ã % _timestamp‚1.7842387756537297e+09 _runtime‚ 717.5176919 _step‚86 ' lr-AdamW/pg1‚0.00023592539599972935 ' lr-AdamW/pg2‚0.00023592539599972935 trainer/global_step‚1099V¶õ8-0¢+ _wandbruntime‚717  _runtime‚717e»À¢Z £·O IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 717.518518681 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897219083264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897219083264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5779411 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579672986353 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19654506 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897307742720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11004891 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83684949663856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 I|c654506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85573995807232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚87 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8250922 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63403583410688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11015033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2471511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760/E€Á'¤»O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579672986353 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19654506 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897307742720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11004891 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83684949663856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85573995807232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚87 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8250922 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63403583410688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11015033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2471511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 717.518518681 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897219083264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897219083264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5779411WÌÃ&è0¥+ _wandbruntime‚717  _runtime‚717øËÓÆ&¦ÀM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19671969 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63459627028992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898004798976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 898093465600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5784463 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2473514 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84654850832239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85650036895232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8257810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 925431293440 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647100 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84751926390272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 717.914919452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64385058322432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85649931189248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647102 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83759223149888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84654747006345 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8647102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11014711 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84751926390272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63459627028992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898004798976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 898093465600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5784463 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2473514 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84654850832239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85650036895232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8257810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 925431293440 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647100 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84751926390272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 717.914919452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64385058322432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85649931189248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647102 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83759223149888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84654747006345 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8647102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11014711 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0XÖ´à0¨+ _wandbruntime‚717  _runtime‚717ËÍ»©ü 0loss/FRt2/assignment_loss‚1.7487091377114408 0loss/FBt2/assignment_loss‚0.3054790495700893 -loss/FRt2/detection_loss‚1.33677334684533 1loss/SRqqt1/detection_loss‚1.9935228208473381 /loss/FBt2/detection_loss‚1.6642616754253288 &loss/total_loss‚15.646392732711199 % _timestamp‚1.7842387760521529e+09 _runtime‚ 717.915778263 1loss/FBt1/assignment_loss‚0.15550265032144536 /loss/FBt1/detection_loss‚1.5872237599718393 1loss/SRqqt2/assignment_loss‚1.007579280965799 /loss/FRt1/detection_loss‚0.6247591961140759 1loss/SRqqt2/detection_loss‚0.7320545115717345 epoch‚2 trainer/global_step‚1099 _step‚89 0loss/FRt1/assignment_loss‚8E>G3.2308884742199826 2loss/SRqqt1/assignment_loss‚1.2596388291467957KÑ ˜ª€ _step‚89 0loss/FRt1/assignment_loss‚3.2308884742199826 2loss/SRqqt1/assignment_loss‚1.2596388291467957 0loss/FRt2/assignment_loss‚1.7487091377114408 0loss/FBt2/assignment_loss‚0.3054790495700893 -loss/FRt2/detection_loss‚1.33677334684533 1loss/SRqqt1/detection_loss‚1.9935228208473381 /loss/FBt2/detection_loss‚1.6642616754253288 &loss/total_loss‚15.646392732711199 % _timestamp‚1.7842387760521529e+09 _runtime‚ 717.915778263 1loss/FBt1/assignment_loss‚0.15550265032144536 /loss/FBt1/detection_loss‚1.5872237599718393 1loss/SRqqt2/assignment_loss‚1.007579280965799 /loss/FRt1/detection_loss‚0.6247591961140759 1loss/SRqqt2/detection_loss‚0.7320545115717345 epoch‚2 trainer/global_step‚1099Yq:½¶k«‚(:à º¥åÒòÇæA) disk./.usagePercent‚74.21107985573082 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125' disk.nvme1c1n1.out‚2741.71923828125 disk.md0.in‚0# proc.memory.rssMB‚ 2887.88671875 network.sent‚ 1.50569e+06 disk.dm-1.in‚0! disk.dm-1.out‚265.60009765625' disk.nvme0c0n1.out‚8059.90283203125% disk.nvme1n1.out‚2741.71923828125 disk.md0.out‚ 18.64453125$ memory_percent‚90.17744064331055' proc.memory.availableMB‚ 1609.328125 disk.rbd0.out‚0% disk.nvme0n1.out‚8059.90283203125 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625% disk./.usageGB‚1139.3256034851074 disk.dm-0.in‚ 4.55859375# disk.dm-0.out‚10495.87744140625* proc.memory.percent‚17.626261711120605 cpu‚25.714552048992484 proc.cpu.threads‚16 network.recv‚242727ÆAÆ£¬‚(:’ º¥åÒµËÙA gpu.0.gpu‚60.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚152.774' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.92466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ëaã­‚(:… É¥åÒåŽÝA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.464' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^g®‚(:Ü É¥åÒÁ“êA network.sent‚ 1.567899e+06 disk.md0.out‚19.3125 disk.dm-0.in‚ 4.55859375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125% disk.nvme0n1.out‚8065.62158203125' disk.nvme1c1n1.out‚2742.66845703125 network.recv‚247047 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚90.17786979675293* proc.memory.percent‚17.626261711120605 cpu‚25.71225300116986 proc.cpu.threads‚16) disk./.usagePercent‚74.21109526111955$ disk./.usageGB‚1139.325839996338! disk.dm-1.out‚265.70947265625 disk.nvme1n1.in‚ 1.6640625% disk.nvme1n1.out‚2742.66845703125# proc.memory.rssMB‚ 2887.88671875 disk.md0.in‚0# disk.dm-0.out‚10501.01416015625' disk.nvme0c0n1.out‚8065.62158203125 disk.nvme1c1n1.in‚ 1.6640625( proc.memory.availableMB‚ 1609.2578125©-#B0¯+ _wandbruntime‚748  _runtime‚748^QTfÅ°¿ _step‚90 & lr-AdamW/pg1‚0.0002358155894791298 & lr-AdamW/pg2‚0.0002358155894791298 trainer/global_step‚1149 % _timestamp‚1.7842388065729861e+09 _runtime‚ 748.436929781Ù]Üêɱà & lr-AdamW/pg1‚0.0002358155894791298 & lr-AdamW/pg2‚0.0002358155894791298 trainer/global_step‚1149 % _timestamp‚1.7842388065729861e+09 _runtime‚ 748.436929781 _step‚90Z¦óÀ‰0²+ _wandbruntime‚748  _runtime‚748Òriû½'³·O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 933978892665 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8609568 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67175664524800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225651200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67175438873600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89370829062656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2577174 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89370829062656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 965194714112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88434260039168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚ 748.441348271 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2577357 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88434260039168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11497743 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66210469810688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 936569023488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784238806573836e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66210375700992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 936657682944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 936569023488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89370934761472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88333541612543 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934065679087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88434277078528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚91 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9020921 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9020921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20518664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6032211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131541504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87399475933456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9020923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 965063172608TFãÁ'´»O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88333541612543 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934065679087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88434277078528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚91 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9020921 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9020921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20518664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6032211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131541504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87399475933456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9020923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 965063172608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 933978892665 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8609568 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67175664524800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225651200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67175438873600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89370829062656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2577174 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89370829062656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 965194714112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88434260039168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚ 748.441348271 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2577357 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88434260039168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11497743 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66210469810688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 936569023488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784238806573836e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66210375700992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 936657682944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 936569023488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89370934761472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0[ÜÅÞL0µ+ _wandbruntime‚748  _runtime‚748½¦@P¶ ¶ÂM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚92 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11517719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934849173799 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937354739200 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚1149 _runtime‚ 748.831849322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8616758 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88408511806753 % _timestamp‚1.7842388069676216e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 965869236224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88509409705472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8616581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6037267 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965998673408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937443405824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20536113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2579318 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9028552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88509426744832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 937354739200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 965869236224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88509409705472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8616581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6037267 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965998673408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937443405824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20536113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2579318 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9028552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88509426744832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 937354739200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934849173799 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937354739200 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚1149 _runtime‚ 748.831849322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8616758 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682\Ʋ¦0¸+ _wandbruntime‚748  _runtime‚748BØŠ¹ý 0loss/FRt1/assignment_loss‚3.0941252200616955 /loss/FRt1/detection_loss‚0.5872187509507961 1loss/SRqqt1/detection_loss‚1.6335361953951422 -loss/FBt1/detection_loss‚1.74399711877805 &loss/total_loss‚14.684219181084659 % _timestamp‚1.7842388069693637e+09 0loss/FRt2/assignment_loss‚1.6342943012714386 2loss/SRqqt1/assignment_loss‚1.2250978398473469 1loss/FBt1/assignment_loss‚0.09073760176583429 0loss/SRqqt2/detection_loss‚0.749591763383946 1loss/FBt2/assignment_loss‚0.19262477159024785 /loss/FBt2/detection_loss‚1.4928883435122233 trainer/global_step‚1149 _runtime‚ 748.833004974 _step‚93 2loss/SRqqt2/assignment_loss‚0.9510726244377165 /loss/FRt2/detection_loss‚1.2890346500902212 epoch‚2ÍêNôº 0loss/FRt2/assignment_loss‚1.6342943012714386 2loss/SRqqt1/assignment_loss‚1.2250978398473469 1loss/FBt1/assignment_loss‚0.09073760176583429 0loss/SRqqt2/detection_loss‚0.749591763383946 1loss/FBt2/assignment_loss‚0.19262477159024785 /loss/FBt2/detection_loss‚1.4928883435122233 trainer/global_step‚1149 _runtime‚ 748.833004974 _step‚93 2loss/SRqqt2/assignment_loss‚0.9510726244377165 /loss/FRt2/detection_loss‚1.2890346500902212 epoch‚2 0loss/FRt1/assignment_loss‚3.0941252200616955 /loss/FRt1/detection_loss‚0.5872187509507961 1loss/SRqqt1/detection_loss‚1.6335361953951422 -loss/FBt1/detection_loss‚1.74399711877805 &loss/total_loss‚14.684219181084659 % _timestamp‚1.7842388069693637e+09]¡¡%»‚(:’ Ø¥åÒúóöA gpu.0.gpu‚66.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.603' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.86766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dk7ch¼‚(:Ý Ø¥åÒÊó‚B$ disk.nvme1n1.out‚2743.4775390625 disk.md0.out‚ 19.7734375 disk.dm-1.in‚0) proc.memory.availableMB‚ 1609.51953125# proc.memory.rssMB‚ 2887.88671875 network.recv‚250376 disk.nvme0c0n1.in‚ 2.89453125 disk.dm-0.in‚ 4.55859375 disk.dm-1.out‚265.7607421875$ memory_percent‚90.17627239227295 cpu‚25.71003272738442 disk.nvme0n1.in‚ 2.89453125# disk.dm-0.out‚10510.15966796875* proc.memory.percent‚17.626261711120605 proc.cpu.threads‚16 network.sent‚ 1.572701e+06) disk./.usagePercent‚74.21109998212577 disk.rbd0.out‚0' disk.nvme0c0n1.out‚8075.00927734375% disk.nvme0n1.out‚8075.00927734375 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚2743.4775390625 disk.md0.in‚0$ disk./.usageGB‚1139.325912475586 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.6640625ï°ú𒽂(:‡ ç¥åÒóÊ÷A gpu.0.gpu‚92.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚217.398' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚72.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.^ÂÆf¾‚(:Û ç¥åÒ؉B disk.dm-0.in‚ 4.55859375# disk.dm-0.out‚11384.76513671875 disk.nvme0c0n1.in‚ 2.89453125# proc.memory.rssMB‚ 2887.88671875 proc.cpu.threads‚16# disk.nvme1n1.out‚3039.529296875 disk.dm-1.out‚265.8466796875% disk.nvme0n1.out‚8654.74462890625 disk.nvme1c1n1.in‚ 1.6640625% disk.nvme1c1n1.out‚3039.529296875$ memory_percent‚90.17620086669922 disk.rbd0.out‚0 network.sent‚ 1.634951e+06 network.recv‚254489 disk.dm-1.in‚0' disk.nvme0c0n1.out‚8654.74462890625 disk.nvme0n1.in‚ 2.89453125 cpu‚25.708229220489812 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 1609.53125* proc.memory.percent‚17.626261711120605) disk./.usagePercent‚74.20809692521958% disk./.usageGB‚1139.2798080444336 disk.md0.out‚ 20.28515625{*gG0¿+ _wandbruntime‚779  _runtime‚779üB(4ÇÀÁ _runtime‚ 779.508385649 _step‚94 ' lr-AdamW/pg1‚0.00023569823731408964 ' lr-AdamW/pg2‚0.00023569823731408964 trainer/global_step‚1199 % _timestamp‚1.7842388376425724e+09(«ŽÀËÁÅ % _timestamp‚1.7842388376425724e+09 _runtime‚ 779.508385649 _step‚94 ' lr-AdamW/pg1‚0.00023569823731408964 ' lr-AdamW/pg2‚0.00023569823731408964 trainer/global_step‚1199^ YJ0Â+ _wandbruntime‚779  _runtime‚779PÚ_ñ¿'ùO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70022793478144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92087202587169 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12000737 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2682887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70023012837888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚ 779.508448079 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11990595 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9402371 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91114002203056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93167768016896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2683064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92191743354368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1005531377152 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92087306412951 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12000737 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69017262100992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 976007623168 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 973304209895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93167662318080 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 975918963712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91113985163696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92191760393728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 $ _timestamp‚1.784238837643216e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92191743354368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9402373 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69017356210688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9402373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 973217423473 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 976007623168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93167768016896 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93167662318080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6285007 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93167768016896 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93167662318080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6285007 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12000737 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2682887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70023012837888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚ 779.508448079 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11990595 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9402371 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91114002203056 TGDeviceStatsMonitor.on_õƒÌòtrain_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93167768016896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2683064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92191743354368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1005531377152 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92087306412951 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12000737 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69017262100992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 976007623168 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 973304209895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93167662318080 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 975918963712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91113985163696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92191760393728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 $ _timestamp‚1.784238837643216e+09_p4G0Å+ _wandbruntime‚779  _runtime‚779ÁûRÌÊ&ÆÄM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1006332435456 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 974000918073 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 976793346048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137825792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12010571 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8975041 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70079964199936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2684978 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9410000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69073493938688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93243703406080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70079732264448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93243597700096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12000415 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91188292728448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92266893020672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92266893020672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92162380433055 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚96 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91188275689088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92266910060032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92162276607161 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12000415 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12010571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 976793346048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92266910060032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006470261248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 974087704607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976704679424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9410002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9410000 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93243703406080 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8975041 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70079964199936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2684978 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9410000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69073493938688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93243703406080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70079732264448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93243597700096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12000415 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91188292728448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92266893020672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92266893020672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92162380433055 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚96 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91188275689088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92266910060032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92162276607161 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12000415 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12010571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 976793346048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92266910060032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006470261248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 974087704607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976704679424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9410002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9410000`jCe0È+ _wandbruntime‚779  _runtime‚779±,^Éý 0loss/FRt1/assignment_loss‚3.1403564968550137 0loss/FRt2/assignment_loss‚1.7765206709991428 1loss/SRqqt1/detection_loss‚1.6437453789011467 _runtime‚ 779.901335782 2loss/SRqqt1/assignment_loss‚1.0944576748970105 &loss/total_loss‚17.464879613904408 epoch‚2 % _timestamp‚1.7842388380376909e+09 _step‚97 0loss/FBt2/assignment_loss‚0.2731528194297267 /loss/FRt1/detection_loss‚0.5727578529583376 1loss/SRqqt2/detection_loss‚0.7564529259672729 /loss/FBt1/detection_loss‚3.4235680636184336 -loss/FBt2/detection_loss‚2.16027671681798 2loss/SRqqt2/assignment_loss‚0.9924056367107749 1loss/FBt1/assignment_loss‚0.15436541149728272 /loss/FRt2/detection_loss‚1.4768199652522844 trainer/global_step‚1199äw Ê 2loss/SRqqt1/assignment_loss‚1.0944576748970105 &loss/total_loss‚17.464879613904408 epoch‚2 % _timestamp‚1.7842388380376909e+09 _step‚97 0loss/FBt2/assignment_loss‚0.2731528194297267 /loss/FRt1/detection_loss‚0.5727578529583376 1loss/SRqqt2/detection_loss‚0.7564529259672729 /loss/FBt1/detection_loss‚3.4235680636184336 -loss/FBt2/detection_loss‚2.16027671681798 2loss/SRqqt2/assignment_loss‚0.9924056367107749 1loss/FBt1/assignment_loss‚0.15436541149728272 /loss/FRt2/detection_loss‚1.4768199652522844 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚3.1403564968550137 0loss/FRt2/assignment_loss‚1.7765206709991428 1loss/SRqqt1/detection_loss‚1.6437453789011467 _runtime‚ 779.901335782aœÑ?œË‚(:‘ ö¥åÒø¾ïA gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚129.92' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.30666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀPªgÌ‚(:Ü ö¥åÒÍ€þA network.recv‚257907( disk./.usagePercent‚74.2081205302507 disk.nvme0n1.in‚ 2.89453125 proc.cpu.threads‚16 network.sent‚ 1.639754e+06& disk.nvme0c0n1.out‚8665.8603515625$ disk.nvme0n1.out‚8665.8603515625$ disk.nvme1n1.out‚3040.5751953125" disk.dm-0.out‚11396.2841796875 disk.dm-1.in‚0 disk.dm-1.out‚265.9443359375$ memory_percent‚90.17693996429443% disk./.usageGB‚1139.2801704406738 disk.nvme0c0n1.in‚ 2.89453125& disk.nvme1c1n1.out‚3040.5751953125 disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 20.52734375) proc.memory.availableMB‚ 1609.41015625# proc.memory.rssMB‚ 2887.88671875 cpu‚25.706503437262683 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.in‚0 disk.dm-0.in‚ 4.55859375 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.626261711120605—G3È0Í+ _wandbruntime‚792  _runtime‚792šÌ%tŒÎð{ 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5332939624786377 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.25554874539375305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006998969707638025 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6313765645027161 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6527777910232544 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.31237924098968506 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9620798230171204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958295226097107 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19858376681804657 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5896695256233215 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5783181190490723 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8244706392288208 _runtime‚ 792.868371541 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5592448115348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9517413377761841 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09396205842494965 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14440660178661346 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.21136829257011414 +particle/specificity‚0.8870100975036621 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09228204190731049 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9795328974723816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0017765199299901724 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19340936839580536 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48977625370025635 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9763075709342957 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_4‚0.5102956295013428 particle/accuracy_3_of_5‚1 % _timestamp‚1.7842388510039933e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14196369051933289 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6088658571243286 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9857087135314941 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8318431377410889 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5524889230728149 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.40666666626930237 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7505790591239929 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7003024220466614 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.28703704476356506 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18149906396865845 *jet/accuracy_3_of_4‚0.0550873689353466 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16977593302726746 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2389688491821289 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22805504500865936 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6374182105064392 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2725656032562256 7validation_average_jet_accuracy‚0.31264784932136536 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21986804902553558 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9425888061523438 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6297306418418884 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,jet/accuracy_4_of_4‚0.006769374944269657 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958295226097107 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03272777050733566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2067607194185257 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9687108397483826 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5533104538917542 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 _step‚98 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5601621270179749 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.28703704476356506 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1873769313097 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08636735379695892 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 particle/accuracy_3_of_4‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.Þ…3(j&9065479040145874 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2530127763748169 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966968297958374 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32642805576324463 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9532738924026489 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13320057094097137 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425925374031067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5707003474235535 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7030801773071289 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.35070183873176575 +particle/sensitivity‚0.5957795977592468 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888917803764343 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2418370544910431 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4606754183769226 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17104710638523102 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5570597648620605 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47068285942077637 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4925622045993805 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47852760553359985 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5884765386581421 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_2_of_2‚0.09749186784029007 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9944025874137878 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003074652748182416 /particle/accuracy_4_of_4‚0.9622277021408081 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5026041865348816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5533104538917542 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14440660178661346 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5597994923591614 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6527777910232544 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.802871823310852 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9827974438667297 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9719954133033752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6303938031196594 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_1‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_3‚1 epoch‚2 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030361592769622803 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5975092053413391 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1382969170808792 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4453037679195404 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_1_of_3‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7952051758766174 )jet/accuracy_2_of_3‚0.375296950340271 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5392954349517822 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4750756323337555 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9827381372451782 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8538282513618469 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924196600914001 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1509466916322708 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42118480801582336 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15874111652374268 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2366463541984558 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4593551754951477 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5979029536247253 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16953125596046448 ,jet/accuracy_3_of_3‚0.035130299627780914 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.531618595123291 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918155670166016 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5606425404548645 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15399819612503052 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5181357264518738 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5190098881721497 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994099497795105 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4444853365421295 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6318290829658508 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4925622045993805 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20432697236537933 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7099395394325256 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6361811757087708 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35185182094573975 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚15.43407079563898 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9828974604606628 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5354304313659668 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013345672748982906 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21136829257011414 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 )jet/accuracy_1_of_2‚0.730128288269043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5329022407531738 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42941224575042725 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_4‚0.8661965727806091 jet/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1116207093000412 trainer/global_step‚1199 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20823340117931366 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5508086681365967 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9793097376823425 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.21235470473766327 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15596777200698853 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24720866978168488 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712433934211731 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47233137488365173 'particle/f_score‚0.5931116938591003 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15990447998046875 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9833450317382812 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24321438372135162 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_3‚0.8417937159538269 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11660353094339371 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5434986352920532 /particle/accuracy_3_of_3‚0.9989922046661377 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11517975479364395 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21399559080600739 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32058805227279663 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8148148059844971K34ú=Ïô{ 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.40666666626930237 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7505790591239929 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7003024220466614 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.28703704476356506 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18149906396865845 *jet/accuracy_3_of_4‚0.0550873689353466 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16977593302726746 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2389688491821289 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22805504500865936 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6374182105064392 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2725656032562256 7validation_average_jet_accuracy‚0.31264784932136536 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21986804902553558 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9425888061523438 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6297306418418884 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,jet/accuracy_4_of_4‚0.006769374944269657 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958295226097107 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03272777050733566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2067607194185257 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9687108397483826 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5533104538917542 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 _step‚98 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5601621270179749 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.28703704476356506 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1873769313097 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08636735379695892 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 particle/accuracy_3_of_4‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9065479040145874 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2530127763748169 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966968297958374 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32642805576324463 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9532738924026489 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13320057094097137 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425925374031067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5707003474235535 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7030801773071289 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.35070183873176575 +particle/sensitivity‚0.5957795977592468 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888917803764343 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2418370544910431 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4606754183769226 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17104710638523102 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5570597648620605 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47068285942077637 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4925622045993805 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47852760553359985 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5884765386581421 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_2_of_2‚0.09749186784029007 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9944025874137878 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003074652748182416 /particle/accuracy_4_of_4‚0.9622277021408081 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5026041865348816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5533104538917542 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14440660178661346 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5597994923591614 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6527777910232544 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.802871823310852 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9827974438667297 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9719954133033752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6303938031196594 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_1‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_3‚1 epoch‚2 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030361592769622803 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5975092053413391 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1382969170808792 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4453037679195404 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_1_of_3‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7952051758766174 )jet/accuracy_2_of_3‚0.375296950340271 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5392954349517822 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4750756323337555 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9827381372451782 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8538282513618469 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924196600914001 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1509466916322708 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42118480801582336 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15874111652374268 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2366463541984558 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4593551754951477 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5979029536247253 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16953125596046448 ,jet/accuracy_3_of_3‚0.035130299627780914 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.531618595123291 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918155670166016 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5606425404548645 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15399819612503052 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5181357264518738 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5190098881721497 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994099497795105 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4444853365421295 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6318290829658508 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4925622045993805 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20432697236537933 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7099395394325256 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6361811757087708 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35185182094573975 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚15.43407079563898 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9828974604606628 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5354304313659668 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013345672748982906 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21136829257011414 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 )jet/accuracy_1_of_2‚0.730128288269043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5329022407531738 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42941224575042725 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_4‚0.8661965727806091 jet/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1116207093000412 trainer/global_step‚1199 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20823340117931366 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5508086681365967 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9793097376823425 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.21235470473766327 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15596777200698853 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24720866978168488 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712433934211731 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47233137488365173 'particle/f_score‚0.5931116938591003 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15990447998046875 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9833450317382812 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24321438372135162 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_3‚0.8417937159538269 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11660353094339371 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5434986352920532 /particle/accuracy_3_of_3‚0.9989922046661377 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11517975479364395 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21399559080600739 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32058805227279663 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8148148059844971 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5332939624786377 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.25554874539375305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006998969707638025 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6313765645027161 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6527777910232544 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.31237924098968506 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9620798230171204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958295226097107 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19858376681804657 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5896695256233215 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5783181190490723 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8244706392288208 _runtime‚ 792.868371541 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5592448115348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9517413377761841 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09396205842494965 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14440660178661346 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.21136829257011414 +particle/specificity‚0.8870100975036621 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09228204190731049 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9795328974723816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0017765199299901724 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19340936839580536 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48977625370025635 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9763075709342957 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_4‚0.5102956295013428 particle/accuracy_3_of_5‚1 % _timestamp‚1.7842388510039933e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14196369051933289 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6088658571243286 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9857087135314941 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8318431377410889 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5524889230728149b&ÌЂ(:’ …¦åÒ­øãA gpu.0.gpu‚86.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚157.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.64233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÒ*hÑ‚(:Ý …¦åÒûƒB proc.cpu.threads‚16 network.sent‚ 1.701056e+06% disk./.usageGB‚1139.2839393615723' disk.nvme1c1n1.out‚3104.99755859375 disk.dm-0.in‚ 4.55859375! proc.memory.rssMB‚ 2910.046875* proc.memory.percent‚17.761516571044922 cpu‚25.754336190022457 disk.nvme1n1.in‚ 1.6640625% disk.nvme1n1.out‚3104.99755859375 disk.md0.in‚0 disk.md0.out‚ 20.78515625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8675.04052734375 disk.nvme0n1.in‚ 2.89453125" memory_percent‚90.702223777771 network.recv‚260969) disk./.usagePercent‚74.20836602257432 disk.nvme1c1n1.in‚ 1.6640625" disk.dm-0.out‚11405.3818359375 disk.dm-1.in‚0 disk.dm-1.out‚329.8701171875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125% disk.nvme0n1.out‚8675.04052734375) proc.memory.availableMB‚ 1523.34765625 Ê‘[hÒ‚(:Ý ”¦åÒÁÁB disk.nvme0c0n1.in‚ 2.89453125 disk.dm-1.in‚0( proc.memory.availableMB‚ 1521.7578125* proc.memory.percent‚17.771315574645996 disk.nvme0n1.in‚ 2.89453125' disk.nvme1c1n1.out‚3184.70458984375 disk.nvme1n1.in‚ 1.6640625# disk.dm-0.out‚12328.90478515625! disk.dm-1.out‚393.79833984375$ memory_percent‚90.71192741394043# proc.memory.rssMB‚ 2911.65234375) disk./.usagePercent‚74.26547429554239 disk.nvme1c1n1.in‚ 1.6640625 disk.rbd0.out‚0 cpu‚25.7524343748139 proc.cpu.threads‚16 disk.md0.out‚21.15625 network.recv‚264767' disk.nvme0c0n1.out‚9583.58447265625% disk.nvme0n1.out‚9583.58447265625% disk.nvme1n1.out‚3184.70458984375 disk.md0.in‚0 disk.dm-0.in‚ 4.55859375 disk.rbd0.in‚0 network.sent‚ 1.758085e+06% disk./.usageGB‚1140.1606941223145ÞÓ‚(:’ ”¦åÒÙýñA gpu.0.gpu‚95.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.016' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.00533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë–5Ÿ0Ô+ _wandbruntime‚824  _runtime‚8242'…’ÄÕ¾ _step‚99 & lr-AdamW/pg1‚0.0002355733470419191 & lr-AdamW/pg2‚0.0002355733470419191 trainer/global_step‚1249 $ _timestamp‚1.784238882267885e+09 _runtime‚ 824.131830965ÏÿÈ&ÈÖ trainer/global_step‚1249 $ _timestamp‚1.784238882267885e+09 _runtime‚ 824.131830965 _step‚99 & lr-AdamW/pg1‚0.0002355733470419191 & lr-AdamW/pg2‚0.0002355733470419191cd‹¾$0×+ _wandbruntime‚824  _runtime‚824·º3YØÁO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95801978469488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22471573 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 114763776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842388822687407e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96945713965568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2836689 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9482751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1027871212842 trainer/global_step‚1249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96945731004928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1030786939904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96945731004928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1030698279936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚164 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚100 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12565952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73689245223936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12576095 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1030698279936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95801995508848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905621 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96829849682330 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97976517944832 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97976412245504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96829953508116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030786939904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97976412245504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72627513705984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97976517944832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1027957999268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72627607815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 _runtime‚ 824.132558617 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12576095 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6646062 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905621 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96945713965568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12565952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1061731517952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96829849682330 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97976517944832 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97976412245504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96829953508116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030786939904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97976412245504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72627513705984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97976517944832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1027957999268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72627607815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 _runtime‚ 824.132558617 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12576095 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0d~üxÎ0Ú+ _wandbruntime‚824  _runtime‚824I8ÄÑ&ÛËM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97020880671232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72683651433984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7842388826631227e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2838900 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95876286034240 _runtime‚ 824.527265381 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22489022 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9490018 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1031483995648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚101 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913250 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97020863631872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1031572662784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12575772 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913252 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97020863631872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2839076 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98052453334016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1031483995648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72683745543680 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98052453334016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1028654707442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6651118 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73746399585792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96904923702322 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12575772 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960ÈkjË£ÜÏM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1028654707442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6651118 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73746399585792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96904923702322 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12575772 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97020880671232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72683651433984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7842388826631227e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2838900 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95876286034240 _runtime‚ 824.527265381 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22489022 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9490018 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1031483995648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚101 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913250 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97020863631872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1031572662784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12575772 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913252 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97020863631872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2839076 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_töÎbó2rain_batch_end/segment.small_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98052453334016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1031483995648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72683745543680 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98052453334016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0eÊf 0Ý+ _wandbruntime‚824  _runtime‚824ÔVb“Þÿ /loss/FRt2/assignment_loss‚1.630810703873241 /loss/FBt1/detection_loss‚1.4882904765505016 epoch‚3 _runtime‚ 824.527789503 2loss/SRqqt1/assignment_loss‚1.0280262723699953 1loss/FBt2/assignment_loss‚0.24661671643803187 1loss/SRqqt2/detection_loss‚0.7676693596071998 % _timestamp‚1.7842388826641495e+09 _step‚102 2loss/SRqqt2/assignment_loss‚0.9986549939725874 /loss/FRt1/detection_loss‚0.6296676633227305 /loss/FRt2/detection_loss‚1.3358929157083195 1loss/SRqqt1/detection_loss‚1.4710175217576789 /loss/FBt2/detection_loss‚1.0747734932212012 trainer/global_step‚1249 /loss/FRt1/assignment_loss‚2.925733644067155 1loss/FBt1/assignment_loss‚0.15066754624104184 &loss/total_loss‚13.747821307129684œÎ» ߃ /loss/FRt2/assignment_loss‚1.630810703873241 /loss/FBt1/detection_loss‚1.4882904765505016 epoch‚3 _runtime‚ 824.527789503 2loss/SRqqt1/assignment_loss‚1.0280262723699953 1loss/FBt2/assignment_loss‚0.24661671643803187 1loss/SRqqt2/detection_loss‚0.7676693596071998 % _timestamp‚1.7842388826641495e+09 _step‚102 2loss/SRqqt2/assignment_loss‚0.9986549939725874 /loss/FRt1/detection_loss‚0.6296676633227305 /loss/FRt2/detection_loss‚1.3358929157083195 1loss/SRqqt1/detection_loss‚1.4710175217576789 /loss/FBt2/detection_loss‚1.0747734932212012 trainer/global_step‚1249 /loss/FRt1/assignment_loss‚2.925733644067155 1loss/FBt1/assignment_loss‚0.15066754624104184 &loss/total_loss‚13.747821307129684f UOmlà‚(:á £¦åÒù¶éA disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125% disk.nvme0n1.out‚9589.99853515625 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 proc.cpu.threads‚16( disk./.usagePercent‚74.0885612961136$ disk./.usageGB‚1137.444637298584 disk.dm-1.in‚0! disk.dm-1.out‚393.85693359375 disk.nvme0n1.in‚ 2.89453125# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 network.recv‚268621 disk.rbd0.in‚0% disk.nvme1n1.out‚3185.66552734375$ memory_percent‚90.71276187896729) proc.memory.availableMB‚ 1521.62109375 network.sent‚ 1.763045e+06 disk.md0.in‚0 disk.md0.out‚ 21.83203125 disk.dm-0.in‚ 4.55859375# disk.dm-0.out‚12334.79931640625' disk.nvme0c0n1.out‚9589.99853515625' disk.nvme1c1n1.out‚3185.66552734375 cpu‚25.7497181863415321œ.œá‚(:‘ £¦åÒ°¯ÜA gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.294' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.431333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nHBõâ‚(:’ ²¦åÒ´òÍA gpu.0.gpu‚88.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚181.004' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.33466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å“Ú¥Yã‚(:Î ²¦åÒ´ÙÜA" disk.nvme0n1.out‚ 9597.46484375) disk./.usagePercent‚74.08851532842142% disk./.usageGB‚1137.4439315795898 disk.dm-1.in‚0 disk.dm-1.out‚394.0068359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 22.02734375# memory_percent‚90.7132625579834( proc.memory.availableMB‚ 1521.5390625 cpu‚25.74738485327611 disk.dm-0.in‚ 4.55859375" disk.dm-0.out‚12342.4716796875 disk.nvme1c1n1.in‚ 1.6640625# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 network.sent‚ 1.825326e+06 network.recv‚272534$ disk.nvme0c0n1.out‚ 9597.46484375" disk.nvme1c1n1.out‚ 3186.453125 disk.nvme1n1.out‚ 3186.453125 disk.md0.in‚0 proc.cpu.threads‚16 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125äqä‚(:’ Á¦åÒ÷‡ÏA gpu.0.gpu‚92.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚146.143' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.71433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üzlcå‚(:Ø Á¦åÒùýßA disk.dm-0.in‚ 4.55859375" disk.dm-0.out‚12348.4638671875% disk.nvme0c0n1.out‚9603.939453125) disk./.usagePercent‚74.08849768676659 disk.nvme1c1n1.in‚ 1.6640625 disk.dm-1.out‚394.0537109375 disk.nvme0c0n1.in‚ 2.89453125 disk.nvme0n1.in‚ 2.89453125$ memory_percent‚90.71409702301025* proc.memory.percent‚17.771315574645996 network.sent‚ 1.830124e+06% disk.nvme1c1n1.out‚3187.240234375 disk.nvme1n1.in‚ 1.6640625 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.745124386591748 network.recv‚275917 disk.md0.out‚ 22.60546875 disk.rbd0.in‚0# disk.nvme0n1.out‚9603.939453125) proc.memory.availableMB‚ 1521.40234375# proc.memory.rssMB‚ 2911.65234375 proc.cpu.threads‚16$ disk./.usageGB‚1137.443660736084# disk.nvme1n1.out‚3187.240234375 disk.md0.in‚0¤ûI 0æ+ _wandbruntime‚855  _runtime‚855¨6ß-ÆçÀ trainer/global_step‚1299 $ _timestamp‚1.784238913468443e+09 _runtime‚ 855.33241395 _step‚103 ' lr-AdamW/pg1‚0.00023544092668408703 ' lr-AdamW/pg2‚0.00023544092668408703*ÃÅÊèÄ ' lr-AdamW/pg1‚0.00023544092668408703 ' lr-AdamW/pg2‚0.00023544092668408703 trainer/global_step‚1299 $ _timestamp‚1.784238913468443e+09 _runtime‚ 855.33241395 _step‚103gtš}0é+ _wandbruntime‚855  _runtime‚855¢‘YÕ'êÏO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287071 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13068947 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101773351200256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101773245500928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100703214320128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2946389 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6898866 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1102500414464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100583718308524 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1067109743650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13058804 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75434400105984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99516504739088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9845255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213067776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6898862 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1070136880128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101773245500928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1102381456384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101773351200256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100703214320128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100703197280768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99516521778448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842389134694386e+09 _step‚104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1070136880128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2946205 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10287071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100583614482738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1070048220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100703197280768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13058804 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚8127¶NÙ'ëÓO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13068947 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101773351200256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101773245500928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100703214320128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2946389 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6898866 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1102500414464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100583718308524 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1067109743650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13058804 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75434400105984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99516504739088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9845255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213067776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6898862 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1070136880128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101773245500928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1102381456384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101773351200256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100703214320128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100703197280768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99516521778448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842389134694386e+09 _step‚104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1070136880128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2946205 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10287071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100583614482738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1070048220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100703197280768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13058804 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287071 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200hĺ‘0ì+ _wandbruntime‚855  _runtime‚855ính©F íÙM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10294700 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9852282 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1070833935872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_‡Öà‹™train_batch_end/requested_bytes.small_pool.freed‚ 1067893238250 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100778363986432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2948364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚1299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1070922603008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10294702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100658688502730 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100778346947072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100658792328628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100778346947072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13068624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1103321806336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103194466816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101849286589440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100778363986432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13078781 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101849286589440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10294700 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99590812303840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101849180882944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10294702 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚105 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99590795264480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23363324 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76593953750016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1067980024788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13078781 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2948547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚105 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99590795264480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23363324 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76593953750016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1067980024788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13078781 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2948547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9852282 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1070833935872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1067893238250 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100778363986432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2948364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚1299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1070922603008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10294702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100658688502730 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100778346947072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100658792328628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100778346947072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13068624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1103321806336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103194466816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101849286589440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100778363986432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13078781 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101849286589440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10294700 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99590812303840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960ik§´0ï+ _wandbruntime‚855  _runtime‚855ªÌ™Eðý _runtime‚ 855.727267404 /loss/FRt2/detection_loss‚1.3958195610032458 1loss/SRqqt1/detection_loss‚1.6861043668453297 % _timestamp‚1.7842389138636065e+09 _step‚106 .loss/FRt1/assignment_loss‚3.06921771219566 /loss/FRt2/assignment_loss‚1.517298174421855 1loss/FBt1/assignment_loss‚0.08971116066848686 /loss/FRt1/detection_loss‚0.5508663749894571 &loss/total_loss‚14.541963054174968 /loss/FBt2/detection_loss‚1.8499792943228826 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.0898481873714405 2loss/SRqqt2/assignment_loss‚0.8821086022025973 0loss/FBt2/assignment_loss‚0.3456591044023023 1loss/SRqqt2/detection_loss‚0.7746157156519011 /loss/FBt1/detection_loss‚1.2907348000998107 epoch‚3&D¦ñ /loss/FBt2/detection_loss‚1.8499792943228826 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.0898481873714405 2loss/SRqqt2/assignment_loss‚0.8821086022025973 0loss/FBt2/assignment_loss‚0.3456591044023023 1loss/SRqqt2/detection_loss‚0.7746157156519011 /loss/FBt1/detection_loss‚1.2907348000998107 epoch‚3 _runtime‚ 855.727267404 /loss/FRt2/detection_loss‚1.3958195610032458 1loss/SRqqt1/detection_loss‚1.6861043668453297 % _timestamp‚1.7842389138636065e+09 _step‚106 .loss/FRt1/assignment_loss‚3.06921771219566 /loss/FRt2/assignment_loss‚1.517298174421855 1loss/FBt1/assignment_loss‚0.08971116066848686 /loss/FRt1/detection_loss‚0.5508663749894571 &loss/total_loss‚14.541963054174968jÔ¦˜­ò‚(:’ ЦåÒý­ÙA gpu.0.gpu‚92.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.905' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.63499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õx§ýdó‚(:Ù ЦåÒðžîA disk.md0.in‚0 disk.md0.out‚ 23.01953125 network.sent‚ 1.835079e+06% disk.nvme0c0n1.out‚9615.189453125 disk.nvme0n1.in‚ 2.89453125 disk.nvme1c1n1.in‚ 1.6640625 disk.dm-0.in‚ 4.55859375 disk.rbd0.in‚0& disk.nvme1c1n1.out‚3189.1982421875" disk.dm-0.out‚12360.6279296875 disk.dm-1.out‚ 394.19140625$ memory_percent‚90.71338176727295) proc.memory.availableMB‚ 1521.51953125* proc.memory.percent‚17.771315574645996 cpu‚25.742652441172375) disk./.usagePercent‚74.08853123075816% disk./.usageGB‚1137.4441757202148 disk.nvme0c0n1.in‚ 2.89453125# disk.nvme0n1.out‚9615.189453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2911.65234375 proc.cpu.threads‚16 network.recv‚279406 disk.nvme1n1.in‚ 1.6640625$ disk.nvme1n1.out‚3189.1982421875 disk.rbd0.out‚0;‹®maô‚(:Ö ߦåÒ„ÍÌA) proc.memory.availableMB‚ 1521.44140625 cpu‚25.740285521557638 disk.dm-1.out‚ 394.2265625 disk.rbd0.out‚0 network.recv‚283813 disk.nvme1n1.in‚ 1.6640625" disk.dm-0.out‚12368.2451171875# disk.nvme0n1.out‚9622.900390625 disk.nvme1c1n1.in‚ 1.6640625& disk.nvme1c1n1.out‚3189.6083984375% disk./.usageGB‚1137.4442176818848 disk.md0.in‚0 disk.nvme0c0n1.in‚ 2.89453125# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 proc.cpu.threads‚16 disk.rbd0.in‚0 network.sent‚ 1.897343e+06) disk./.usagePercent‚74.08853396397231$ disk.nvme1n1.out‚3189.6083984375 disk.dm-0.in‚ 4.55859375 disk.dm-1.in‚0% disk.nvme0c0n1.out‚9622.900390625 disk.nvme0n1.in‚ 2.89453125 disk.md0.out‚ 23.234375$ memory_percent‚90.71385860443115€ˆ¨žõ‚(:“ ߦåÒïðÀA gpu.0.gpu‚44.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.642' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.547333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BýÌ80ö+ _wandbruntime‚886  _runtime‚886f’Ç\È÷ ' lr-AdamW/pg1‚0.00023530098474570555 ' lr-AdamW/pg2‚0.00023530098474570555 trainer/global_step‚1349 % _timestamp‚1.7842389441687527e+09 _runtime‚ 886.032696963 _step‚107L5šhÌøÆ _runtime‚ 886.032696963 _step‚107 ' lr-AdamW/pg1‚0.00023530098474570555 ' lr-AdamW/pg2‚0.00023530098474570555 trainer/global_step‚1349 % _timestamp‚1.7842389441687527e+09k’œø0ù+ _wandbruntime‚886  _runtime‚886RŒäÙ'úÓO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104460680595968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10668521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104460697635328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10668523 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13551656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104460697635328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7151666 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103231048048048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103231031008688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7151662 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1106348274458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10668521 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79384225653248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13561799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104337379283146 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10205769 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78241380615680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚ 886.036327638 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10668523 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105570078756352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104460680595968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1109486820352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1109486820352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105570184455680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1142939147264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104337483108932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1109398160384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13551656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79384447109632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105570078756352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 10¥àOVÉ%3825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103231048048048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103231031008688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7151662 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1106348274458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10668521 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79384225653248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13561799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104337379283146 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10205769 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78241380615680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚ 886.036327638 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10668523 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105570078756352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104460680595968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1109486820352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1109486820352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105570184455680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1142939147264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104337483108932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1109398160384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13551656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79384447109632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105570078756352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13571633 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1110183876096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13561476 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103305321534080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3056298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1107218555596 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10213020 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1110272543232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10212842 _step‚109 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104535847301632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79441173090304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 886.426543714 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24247785 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1143872001536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78297518343680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103305338573440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 $ _timestamp‚1.784238944562442e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104412557129036 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1143748856320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105646014138368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79441390345216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105646119844864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105646119844864|˜P…æ&þàM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78297518343680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103305338573440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 $ _timestamp‚1.784238944562442e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104412557129036 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1143748856320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105646014138368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79441390345216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105646119844864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105646119844864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7156722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104535847301632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78297424233984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104412453303138 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13571633 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1110183876096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13561476 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103305321534080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3056298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1107218555596 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10213020 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1110272543232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10212842 _step‚109 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104535847301632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79441173090304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 886.426543714 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24247785 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1143872001536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0m¡Ÿ¬0ÿ+ _wandbruntime‚886  _runtime‚886dŽFF€þ 2loss/SRqqt1/assignment_loss‚0.8931274595844156 /loss/FRt1/detection_loss‚0.5939499187402666 /loss/FRt2/detection_loss‚1.3874624353153817 trainer/global_step‚1349 /loss/FRt1/assignment_loss‚3.109693683557305 2loss/SRqqt2/assignment_loss‚0.8542870018791752 1loss/FBt2/assignment_loss‚0.34721623125735246 %loss/total_loss‚13.72104837541061 epoch‚3 0loss/FRt2/assignment_loss‚1.5598407570135633 1loss/FBt1/assignment_loss‚0.13435976893466253 1loss/SRqqt1/detection_loss‚1.4835268757288822 0loss/SRqqt2/detection_loss‚0.767013921193098 /loss/FBt2/detection_loss‚1.1563506252490556 _runtime‚ 886.426978994 _step‚110 /loss/FBt1/detection_loss‚1.4342196969574512 % _timestamp‚1.7842389445633795e+09þ 3ׂ 0loss/FRt2/assignment_loss‚1.5598407570135633 1loss/FBt1/assignment_loss‚0.13435976893466253 1loss/SRqqt1/detection_loss‚1.4835268757288822 0loss/SRqqt2/detection_loss‚0.767013921193098 /loss/FBt2/detection_loss‚1.1563506252490556 _runtime‚ 886.426978994 _step‚110 /loss/FBt1/detection_loss‚1.4342196969574512 % _timestamp‚1.7842389445633795e+09 2loss/SRqqt1/assignment_loss‚0.8931274595844156 /loss/FRt1/detection_loss‚0.5939499187402666 /loss/FRt2/detection_loss‚1.3874624353153817 trainer/global_step‚1349 /loss/FRt1/assignment_loss‚3.109693683557305 2loss/SRqqt2/assignment_loss‚0.8542870018791752 1loss/FBt2/assignment_loss‚0.34721623125735246 %loss/total_loss‚13.72104837541061 epoch‚3nÊ­£ÚŽ‚‚(:ƒ î¦åÒõëÔA gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚220.47' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.49 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨î´jƒ‚(:ß î¦åÒ‰ŸâA network.recv‚287123% disk./.usageGB‚1134.7313346862793 disk.nvme1c1n1.in‚ 1.6640625' disk.nvme1c1n1.out‚3190.20751953125 disk.nvme1n1.in‚ 1.6640625 disk.dm-0.in‚ 4.55859375 disk.nvme0n1.in‚ 2.89453125( disk./.usagePercent‚73.9118276949213% disk.nvme1n1.out‚3190.20751953125# disk.dm-0.out‚12379.05712890625 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚9633.67724609375# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125( proc.memory.availableMB‚ 1521.4609375 proc.cpu.threads‚16 network.sent‚ 1.902147e+06 disk.md0.out‚ 23.44140625! disk.dm-1.out‚394.32958984375' disk.nvme0c0n1.out‚9633.67724609375# memory_percent‚90.7137393951416 cpu‚25.73851840794146® Å„‚(:“ ý¦åÒù­ÌA gpu.0.gpu‚64.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚²ï‹¸Ù113.657' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.885666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò–MNk…‚(:à ý¦åÒó‘àA disk.nvme1n1.in‚ 1.6640625% disk.nvme1n1.out‚3190.83544921875 network.sent‚ 1.964464e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 2.89453125% disk.nvme0n1.out‚9638.87158203125! disk.dm-1.out‚394.37646484375$ memory_percent‚91.10560417175293) disk./.usagePercent‚73.91185726332871$ disk./.usageGB‚1134.731788635254 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.out‚ 23.79296875 disk.dm-0.in‚ 4.55859375# disk.dm-0.out‚12384.06103515625 disk.dm-1.in‚0( proc.memory.availableMB‚ 1457.2578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125' disk.nvme0c0n1.out‚9638.87158203125' disk.nvme1c1n1.out‚3190.83544921875 cpu‚25.73628063361415 proc.cpu.threads‚16 network.recv‚291377 disk.md0.in‚0# proc.memory.rssMB‚ 2911.44140625* proc.memory.percent‚17.770028114318848Q¹$0†+ _wandbruntime‚917  _runtime‚917SÑ4}ƇÀ & lr-AdamW/pg1‚0.0002351535302149839 & lr-AdamW/pg2‚0.0002351535302149839 trainer/global_step‚1399 % _timestamp‚1.7842389751610453e+09 _runtime‚ 917.026365128 _step‚111žœ.óÊˆÄ trainer/global_step‚1399 % _timestamp‚1.7842389751610453e+09 _runtime‚ 917.026365128 _step‚111 & lr-AdamW/pg1‚0.0002351535302149839 & lr-AdamW/pg2‚0.0002351535302149839oØ 0‰+ _wandbruntime‚917  _runtime‚917±= Ù'ŠÓO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7404466 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82231902325248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11049971 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1145586805266 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1148748100608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3161722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108091144083554 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106945574317648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1145673591692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82231676674560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183635309568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108091247909340 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14054651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109367017711104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7404462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 % _timestamp‚1.7842389751616957e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1148836760576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108218163911168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10566369 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109366912011776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14054651 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106945557278288;ýqÝ'‹×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1148748100608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3161722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108091144083554 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106945574317648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1145673591692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82231676674560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183635309568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108091247909340 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14054651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109367017711104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7404462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 % _timestamp‚1.7842389751616957e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1148836760576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108218163911168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10566369 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109366912011776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14054651 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106945557278288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7404466 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82231902325248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11049971 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1145586805266p1øÌ 0Œ+ _wandbruntime‚917  _runtime‚917&—½â&ÜM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1149622483456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81104310633984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108166218103546 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107019847803680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14064485 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11057600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82288629108736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7409522 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1149533816320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7409518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 917.416601457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1146370299866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14054328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1149533816320 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14064485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10573452 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 _step‚113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108293330616832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11057602 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108166321929444 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108293330616832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82288846363648 trainer/global_step‚1399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109442847393792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3164107 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1146457086404 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7409522 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1149533816320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7409518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 917.416601457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1146370299866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14054328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1149533816320 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14064485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10573452 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 _step‚113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108293330616832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11057602 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108166321929444 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108293330616832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82288846363648 trainer/global_step‚1399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109442847393792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3164107 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1146457086404 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11431423 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111975664265728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110660083547888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111975647226368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113163745267200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11431421 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113163745267200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10926411 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85079333179392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14547503 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85079113820160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1224054514176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚177 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7657262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184912122500 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _step‚116 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1224179763712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111975647226368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110660100587248 % _timestamp‚1.7842390059952698e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11431421 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10926588 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7657266 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1188186700800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113163850966528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83855059305984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1188098040832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14547503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚1449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1188186700800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111844908883962 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14537360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111845012709748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113163745267200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10926411 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85079333179392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14547503 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85079113820160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1224054514176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚177 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7657262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184912122500 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _step‚116 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1224179763712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111975647226368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110660100587248 % _timestamp‚1.7842390059952698e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11431421 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10926588 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7657266 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1188186700800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113163850966528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83855059305984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1188098040832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14547503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚1449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1188186700800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111844908883962 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14537360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111845012709748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11431423 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303t!þ4¯0œ+ _wandbruntime‚948  _runtime‚948+1šâ&ÜM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188972423680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439052 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7662318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83911197033984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10933622 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85136286007808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14547180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3271474 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112050813932032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188883756544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185608830674 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14557337 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7662322 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111920086729852 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14557337 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85136062461440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 $ _timestamp‚1.784239006384557e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14547180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112050796892672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1224865427456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110734374073280 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188972423680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110734391112640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83911291143680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 948.248679637 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112050796892672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113239786355712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1224994864128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3271304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10933796 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188883756544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113239786355712 trainer/global_step‚1449 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439050 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113239680649216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185695617212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 _step‚117éÖ“æ&žàM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113239786355712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1224994864128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3271304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10933796 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188883756544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113239786355712 trainer/global_step‚1449 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439050 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113239680649216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185695617212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 _step‚117 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188972423680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439052 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7662318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83911197033984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10933622 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85136286007808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14547180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3271474 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112050813932032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188883756544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185608830674 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14557337 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7662322 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111920086729852 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14557337 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85136062461440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 $ _timestamp‚1.784239006384557e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14547180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112050796892672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1224865427456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110734374073280 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188972423680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110734391112640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83911291143680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 948.248679637 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112050796892672uŽã¿0Ÿ+ _wandbruntime‚948  _runtime‚948±{Z ÿ 1loss/FBt1/assignment_loss‚0.19679402980258123 /loss/FRt1/detection_loss‚0.5422676645744635 1loss/SRqqt2/detection_loss‚0.7446575142609672 _runtime‚ 948.249141768 2loss/SRqqt1/assignment_loss‚0.9190004599732678 2loss/SRqqt2/assignment_loss‚0.8510290225399955 /loss/FBt1/detection_loss‚1.8348440836106625 /loss/FBt2/detection_loss‚1.2947019593859939 &loss/total_loss‚13.792342227846895 trainer/global_step‚1449 $ _timestamp‚1.784239006385508e+09 _step‚118 /loss/FRt2/detection_loss‚1.4125117047649576 1loss/SRqqt1/detection_loss‚1.3542800065773932 epoch‚3 0loss/FRt1/assignment_loss‚2.9472231402975653 0loss/FRt2/assignment_loss‚1.4591035644552406 0loss/FBt2/assignment_loss‚0.2359290776038068šuÛ ¡ƒ 0loss/FRt1/assignment_loss‚2.9472231402975653 0loss/FRt2/assignment_loss‚1.4591035644552406 0loss/FBt2/assignment_loss‚0.2359290776038068 1loss/FBt1/assignment_loss‚0.19679402980258123 /loss/FRt1/detection_loss‚0.5422676645744635 1loss/SRqqt2/detection_loss‚0.7446575142609672 _runtime‚ 948.249141768 2loss/SRqqt1/assignment_loss‚0.9190004599732678 2loss/SRqqt2/assignment_loss‚0.8510290225399955 /loss/FBt1/detection_loss‚1.8348440836106625 /loss/FBt2/detection_loss‚1.2947019593859939 &loss/total_loss‚13.792342227846895 trainer/global_step‚1449 $ _timestamp‚1.784239006385508e+09 _step‚118 /loss/FRt2/detection_loss‚1.4125117047649576 1loss/SRqqt1/detection_loss‚1.3542800065773932 epoch‚3v’Q]œ¢‚(:‘ ª§åÒòß÷A gpu.0.gpu‚60.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚123.95' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.31666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»`5i£‚(:Þ ª§åÒýø†B disk.dm-0.in‚ 4.55859375 disk.dm-1.in‚0 network.recv‚302226% disk./.usageGB‚1135.7347259521484! disk.dm-1.out‚394.70751953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.89453125& disk.nvme0c0n1.out‚9670.3427734375# memory_percent‚91.1034345626831) disk./.usagePercent‚73.97718455965794% disk.nvme1n1.out‚3193.73291015625" disk.dm-0.out‚12415.3408203125 disk.nvme0n1.in‚ 2.89453125) proc.memory.availableMB‚ 1457.61328125* proc.memory.percent‚17.771315574645996 cpu‚25.730989004082666 proc.cpu.threads‚16 network.sent‚ 2.036048e+06' disk.nvme1c1n1.out‚3193.73291015625 disk.nvme1n1.in‚ 1.6640625$ disk.nvme0n1.out‚9670.3427734375# proc.memory.rssMB‚ 2911.65234375 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.in‚0 disk.md0.out‚ 25.0703125Y¢Ël¤‚(:á ¹§åÒõ«àA disk.dm-0.in‚ 4.55859375 disk.nvme0c0n1.in‚ 2.89453125' disk.nvme0c0n1.out‚11002.3759765625% disk.nvme0n1.out‚11002.3759765625 disk.nvme1c1n1.in‚ 1.6640625' disk.nvme1c1n1.out‚3194.31689453125 disk.nvme1n1.in‚ 1.6640625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1457.84765625 cpu‚25.729016765026227 proc.cpu.threads‚16 network.sent‚ 2.099139e+06 disk.nvme0n1.in‚ 2.89453125% disk.nvme1n1.out‚3194.31689453125# memory_percent‚91.1020040512085# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 network.recv‚306880) disk./.usagePercent‚73.89779661595094 disk.md0.in‚0 disk.md0.out‚ 25.44921875" disk.dm-0.out‚13747.0830078125 disk.dm-1.in‚0! disk.dm-1.out‚394.75830078125 disk.rbd0.in‚0% disk./.usageGB‚1134.5159225463867V³kW’¥‚(:‡ ¹§åÒ´ÞB gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.195' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ñ&´0¦+ _wandbruntime‚977  _runtime‚977‰Ôiȧ ' lr-AdamW/pg1‚0.00023483612174134724 ' lr-AdamW/pg2‚0.00023483612174134724 trainer/global_step‚1499 % _timestamp‚1.7842390361239254e+09 _runtime‚ 977.987868141 _step‚119×\Ì¨Æ ' lr-AdamW/pg1‚0.00023483612174134724 ' lr-AdamW/pg2‚0.00023483612174134724 trainer/global_step‚1499 % _timestamp‚1.7842390361239254e+09 _runtime‚ 977.987868141 _step‚119wø°Œ0©+ _wandbruntime‚977  _runtime‚9775´Ó‰èªÓO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7842390361247566e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 _runtime‚ 977.992324598 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11812873 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86662039815680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115598673684370 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114374609817488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1264694892544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11812871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3376783 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231942144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115733130541568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86661945705984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114374626856848 _step‚120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115733147580928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1224063866882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7910062 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87926734708224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264557060096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7910066 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15040355 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15030212 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87926502766080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116960578522624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116960578522624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3376598 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1227447981056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1227536641024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 trainer/global_step‚1499 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115733130541568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11812871 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11286849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1224150653308 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374ÀG¯úÝ'«×O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7842390361247566e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 _runtime‚ 977.992324598 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11812873 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86662039815680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115598673684370 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114374609817488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1264694892544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11812871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3376783 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231942144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115733130541568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86661945705984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114374626856848 _step‚120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115733147580928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1224063866882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7910062 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87926734708224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264557060096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7910066 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15040355 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15030212 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87926502766080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116960578522624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116960578522624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3376598 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1227447981056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1227536641024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 trainer/global_step‚1499 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115733130541568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11812871 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11286849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1224150653308xÞH0¬+ _wandbruntime‚978  _runtime‚978ó’,ã&­ÝM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11294025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15040032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11820502 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1228322363904 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115808280207872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15050189 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86718083433984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115808297247232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1228322363904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11293834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86718177543680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115673851530260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1224934148020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117036619611136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26860532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1228233696768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7915118 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11820502Á<â®ü!®áM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115808280207872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15050189 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86718083433984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115808297247232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1228322363904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11293834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86718177543680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115673851530260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1224934148020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117036619611136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26860532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1228233696768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7915118 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11820502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11294025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15040032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11820502 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1228322363904 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499y¸Uó0¯+ _wandbruntime‚978  _runtime‚978D9V°ý /loss/FRt1/assignment_loss‚2.928869106656669 2loss/SRqqt2/assignment_loss‚0.7836489660880739 /loss/FBt1/detection_loss‚2.1788835750155213 &loss/total_loss‚14.542724450196854 trainer/global_step‚1499 0loss/FRt2/assignment_loss‚1.4554428529816787 2loss/SRqqt1/assignment_loss‚0.8145038872371515 1loss/FBt1/assignment_loss‚0.11738085524553243 1loss/FBt2/assignment_loss‚0.29128960564910467 -loss/FRt2/detection_loss‚1.29356222088087 /loss/FBt2/detection_loss‚2.1820922096570334 epoch‚3 % _timestamp‚1.7842390365138106e+09 /loss/FRt1/detection_loss‚0.5787603413416971 1loss/SRqqt1/detection_loss‚1.1820404738373327 _runtime‚ 978.37745835 1loss/SRqqt2/detection_loss‚0.7362503556061888 _step‚122B¥µ)± 1loss/SRqqt2/detection_loss‚0.7362503556061888 _step‚122 /loss/FRt1/assignment_loss‚2.928869106656669 2loss/SRqqt2/assignment_loss‚0.7836489660880739 /loss/FBt1/detection_loss‚2.1788835750155213 &loss/total_loss‚14.542724450196854 trainer/global_step‚1499 0loss/FRt2/assignment_loss‚1.4554428529816787 2loss/SRqqt1/assignment_loss‚0.8145038872371515 1loss/FBt1/assignment_loss‚0.11738085524553243 1loss/FBt2/assignment_loss‚0.29128960564910467 -loss/FRt2/detection_loss‚1.29356222088087 /loss/FBt2/detection_loss‚2.1820922096570334 epoch‚3 % _timestamp‚1.7842390365138106e+09 /loss/FRt1/detection_loss‚0.5787603413416971 1loss/SRqqt1/detection_loss‚1.1820404738373327 _runtime‚ 978.37745835zsˆjm²‚(:â ȧåÒÚÜãA) disk./.usagePercent‚74.01218485742919 disk.dm-0.in‚ 4.58203125# disk.dm-0.out‚15340.93701171875 disk.dm-1.in‚0$ memory_percent‚91.10422134399414# proc.memory.rssMB‚ 2911.65234375 cpu‚25.72758457794957 network.sent‚ 2.103462e+06 network.recv‚310256% disk./.usageGB‚1136.2720680236816% disk.nvme1n1.out‚4347.35693359375! disk.dm-1.out‚394.88623046875 disk.nvme0c0n1.in‚ 2.91796875 disk.nvme0n1.in‚ 2.91796875' proc.memory.availableMB‚ 1457.484375 disk.nvme1c1n1.in‚ 1.6640625' disk.nvme1c1n1.out‚4347.35693359375 disk.nvme1n1.in‚ 1.6640625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11444.05615234375& disk.nvme0n1.out‚11444.05615234375* proc.memory.percent‚17.771315574645996 disk.md0.in‚0 disk.md0.out‚ 25.80078125 proc.cpu.threads‚16ŸšÞ³‚(:’ ȧåÒþìØA gpu.0.gpu‚81.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚116.495' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.83166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ùÅ\´‚(:’ קåÒÀþõA gpu.0.gpu‚80.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.609' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.86966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Αñiµ‚(:Þ קåÒª €B$ disk./.usageGB‚1136.272201538086 disk.nvme0n1.in‚ 2.91796875' disk.nvme1c1n1.out‚4455.93994140625 disk.nvme1n1.in‚ 1.6640625% disk.nvme1n1.out‚4455.93994140625 disk.md0.out‚26.1875# disk.dm-0.out‚16192.07177734375( disk.nvme0c0n1.out‚12187.50048828125 network.sent‚ 2.166282e+06 network.recv‚314726( disk./.usagePercent‚74.0121935540196 disk.md0.in‚0 disk.dm-0.in‚ 4.58203125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.6640625 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1457.94921875 cpu‚25.72593845677614 proc.cpu.threads‚16& disk.nvme0n1.out‚12187.50048828125! disk.dm-1.out‚394.94091796875 disk.nvme0c0n1.in‚ 2.91796875$ memory_percent‚91.10138416290283# proc.memory.rssMB‚ 2911.43359375* proc.memory.percent‚17.769980430603027*\S#2¶- _wandbruntime‚1008  _runtime‚1008˜{¸ÌÈ·Â ' lr-AdamW/pg1‚0.00023466618818498538 ' lr-AdamW/pg2‚0.00023466618818498538 trainer/global_step‚1549 $ _timestamp‚1.784239066774606e+09 _runtime‚1008.638581886 _step‚123±g¾IÌ¸Æ ' lr-AdamW/pg1‚0.00023466618818498538 ' lr-AdamW/pg2‚0.00023466618818498538 trainer/global_step‚1549 $ _timestamp‚1.784239066774606e+09 _runtime‚1008.638581886 _step‚123{ÏðÞ˜2¹- _wandbruntime‚1008  _runtime‚1008‹öÚ'ºÔO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 % _timestamp‚1.7842390667754736e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120757411778048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119490613856768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27717385 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1266886581248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3484372 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8162862 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484205 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15533207 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89468926215680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1263389184116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1266797921280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118089153126448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1266886581248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1266797921280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120757411778048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119490630896128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120757517477376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12194323 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118089136087088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120757517477376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12194323 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15523064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12194321 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15523064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119490630896128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11647238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1263302397690 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90773936740864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120757411778048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119490613856768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27717385 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1266886581248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3484372 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8162862 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484205 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15533207 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89468926215680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1263389184116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1266797921280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118089153126448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1266886581248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1266797921280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120757411778048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119490630896128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120757517477376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12194323 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118089136087088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120757517477376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12194323 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15523064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12194321 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15523064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119490630896128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11647238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1263302397690 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90773936740864 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119427616330668 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118163443651840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1267672304128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120833347160064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15543041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119565780562432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120833452866560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89525063943680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1264085892290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚1549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1305912115712 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119565780562432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119565763523072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267583636992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119427512504770 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15532884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3486310 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11654392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1009.028269485 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267583636992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1306039455232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11654228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1264172678828 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3486470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696,Ÿ\ìè&¾âM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267583636992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119427512504770 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15532884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3486310 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11654392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1009.028269485 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267583636992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1306039455232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11654228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1264172678828 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3486470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12201950 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27734834 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119427616330668 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118163443651840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1267672304128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120833347160064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15543041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119565780562432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120833452866560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89525063943680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1264085892290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚1549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1305912115712 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119565780562432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119565763523072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008}÷Œ2¿- _wandbruntime‚1009  _runtime‚1009(Ãj”Àÿ 2loss/SRqqt1/assignment_loss‚0.7893167691778132 /loss/FBt1/detection_loss‚1.5573385041326768 trainer/global_step‚1549 0loss/FRt1/assignment_loss‚2.7892443131441356 0loss/FRt2/assignment_loss‚1.5459245512037412 1loss/FBt2/assignment_loss‚0.06484527573047671 1loss/SRqqt1/detection_loss‚1.3917568025702458 .loss/FBt2/detection_loss‚5.910043607422267 % _timestamp‚1.7842390671650908e+09 _runtime‚1009.028708886 2loss/SRqqt2/assignment_loss‚0.8587874853805543 /loss/FRt1/detection_loss‚0.6430163127999089 /loss/FRt2/detection_loss‚1.5177082984756705 1loss/SRqqt2/detection_loss‚0.8396896326420685 epoch‚3 /loss/FBt1/assignment_loss‚0.099210027555467 &loss/total_loss‚18.006881580235024 _step‚126n9 ø Áƒ _step‚126 2loss/SRqqt1/assignment_loss‚0.7893167691778132 /loss/FBt1/detection_loss‚1.5573385041326768 trainer/global_step‚1549 0loss/FRt1/assignment_loss‚2.7892443131441356 0loss/FRt2/assignment_loss‚1.5459245512037412 1loss/FBt2/assignment_loss‚0.06484527573047671 1loss/SRqqt1/detection_loss‚1.3917568025702458 .loss/FBt2/detection_loss‚5.910043607422267 % _timestamp‚1.7842390671650908e+09 _runtime‚1009.028708886 2loss/SRqqt2/assignment_loss‚0.8587874853805543 /loss/FRt1/detection_loss‚0.6430163127999089 /loss/FRt2/detection_loss‚1.5177082984756705 1loss/SRqqt2/detection_loss‚0.8396896326420685 epoch‚3 /loss/FBt1/assignment_loss‚0.099210027555467 &loss/total_loss‚18.006881580235024~ÌIóM_‚(:Ô æ§åÒøÔâA& disk.nvme0n1.out‚12207.70166015625! disk.nvme1c1n1.out‚ 4456.96875 cpu‚25.724363865034842# disk.dm-0.out‚16212.61962890625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 2.91796875( disk.nvme0c0n1.out‚12207.70166015625 disk.nvme1n1.out‚ 4456.96875( proc.memory.availableMB‚ 1457.6953125% disk./.usageGB‚1133.5549125671387 disk.dm-1.in‚0 disk.dm-1.out‚ 395.03515625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 2.91796875 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0 network.sent‚ 2.17124e+06 network.recv‚318182 disk.dm-0.in‚ 4.58203125 disk.md0.out‚ 26.45703125$ memory_percent‚91.10293388366699# proc.memory.rssMB‚ 2911.65234375* proc.memory.percent‚17.771315574645996 proc.cpu.threads‚16) disk./.usagePercent‚73.83520029748502@ëø„œÃ‚(:‘ æ§åÒ¥õ”B gpu.0.gpu‚84.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚213.56' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.18666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X"¢eÄ‚(:Ú õ§åÒ‚ÚÿA disk.nvme1n1.in‚ 1.6640625 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2911.65234375 cpu‚25.723287861307607 network.sent‚ 2.233156e+06 disk.nvme0n1.in‚ 2.91796875& disk.nvme1c1n1.out‚4489.1806640625 disk.md0.in‚0 disk.md0.out‚ 26.7421875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 2.91796875( disk.nvme0c0n1.out‚12277.04638671875 network.recv‚322646$ disk.nvme1n1.out‚4489.1806640625# disk.dm-0.out‚16313.52978515625( proc.memory.availableMB‚ 1457.4765625* proc.memory.percent‚17.771315574645996 proc.cpu.threads‚16& disk.nvme0n1.out‚12277.04638671875 disk.dm-0.in‚ 4.58203125 disk.dm-1.out‚395.0625$ memory_percent‚91.10426902770996) disk./.usagePercent‚73.83449363739555% disk./.usageGB‚1133.5440635681152 disk.nvme1c1n1.in‚ 1.6640625¿mk’Å‚(:‡ õ§åÒ¾ÄôA gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚211.506' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚70.502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ŸÝ2Æ- _wandbruntime‚1039  _runtime‚1039G°ŠÒÉÇà ' lr-AdamW/pg1‚0.00023448878280808022 ' lr-AdamW/pg2‚0.00023448878280808022 trainer/global_step‚1599 % _timestamp‚1.7842390974288418e+09 _runtime‚1039.292799183 _step‚127É‹+‘ÍÈÇ ' lr-AdamW/pg1‚0.00023448878280808022 ' lr-AdamW/pg2‚0.00023448878280808022 trainer/global_step‚1599 % _timestamp‚1.7842390974288418e+09 _runtime‚1039.292799183 _step‚127Î3P§2É- _wandbruntime‚1039  _runtime‚1039‡fuƒÙ'ÊÓO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123248114211328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3590910 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92275812615680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123248114211328 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12575771 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121803679396048 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123106203285186 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124554245033472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124554350732800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1302540928498 _step‚128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93621453148672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123106307110972 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121803662356688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123248097171968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12006392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92275718505984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1306147861504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123248097171968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12575773 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12575771 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1306147861504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1302627714924 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8415662 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1306236521472 _runtime‚ 1039.29736674 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93621231692288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12575773 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1599 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16015916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16015916 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12006576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345513186304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1306236521472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16026059 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3590730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1345640532992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842390974297175e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124554350732800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124554245033472 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93621231692288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12575773 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1599 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16015916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16015916 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12006576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345513186304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1306236521472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16026059 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3590730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 13456 u&ˆŸ40532992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842390974297175e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124554350732800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124554245033472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124554245033472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124554350732800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1302540928498 _step‚128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93621453148672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123106307110972 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121803662356688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123248097171968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12006392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92275718505984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1306147861504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123248097171968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12575773 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12575771 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1306147861504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1302627714924 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812€Rªûx2Ì- _wandbruntime‚1039  _runtime‚1039q;‚àä&ÍÞM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1346452487680 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28609136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1306933577216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8420722 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93678175090688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16035893 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123323263877632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16025736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12013718 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124630286121984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12583400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92331950343680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1303411209636 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92331856233984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123181277305178 trainer/global_step‚1599 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1307022244352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592996 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123181381131076 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12583400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1306933577216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1307022244352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123323263877632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123323246838272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16025736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93678402831360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121877969921440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124630286121984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12013543 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121877952882080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12583402 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812­(ãCé&ÎãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12583400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1306933577216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1307022244352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123323263877632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123323246838272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16025736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93678402831360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121877969921440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124630286121984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12013543 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121877952882080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12583402 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1346452487680 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28609136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1306933577216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8420722 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93678175090688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16035893 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123323263877632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16025736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12013718 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124630286121984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12583400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92331950343680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1303411209636 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92331856233984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123181277305178 trainer/global_step‚1599 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1307022244352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592996 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123181381131076&Ýb22Ï- _wandbruntime‚1039  _runtime‚1039é×Ðþ /loss/FBt1/detection_loss‚1.0880296257680704 0loss/FRt2/assignment_loss‚1.4725430079692385 2loss/SRqqt1/assignment_loss‚0.9062691501504911 0loss/SRqqt1/detection_loss‚1.129689522515775 1loss/SRqqt2/detection_loss‚0.8297473422025283 trainer/global_step‚1599 % _timestamp‚1.7842390978218555e+09 _runtime‚1039.685507276 1loss/SRqqt2/assignment_loss‚0.736806506935025 0loss/FBt1/assignment_loss‚0.1004693452061963 /loss/FBt2/detection_loss‚0.5685510150023869 epoch‚3 _step‚130 /loss/FRt1/assignment_loss‚2.872001321440519 &loss/total_loss‚11.989392376854644 1loss/FBt2/assignment_loss‚0.26564313943422996 /loss/FRt1/detection_loss‚0.6280118773998639 /loss/FRt2/detection_loss‚1.3916305228303198x1Œ у /loss/FRt2/detection_loss‚1.3916305228303198 /loss/FBt1/detection_loss‚1.0880296257680704 0loss/FRt2/assignment_loss‚1.4725430079692385 2loss/SRqqt1/assignment_loss‚0.9062691501504911 0loss/SRqqt1/detection_loss‚1.129689522515775 1loss/SRqqt2/detection_loss‚0.8297473422025283 trainer/global_step‚1599 % _timestamp‚1.7842390978218555e+09 _runtime‚1039.685507276 1loss/SRqqt2/assignment_loss‚0.736806506935025 0loss/FBt1/assignment_loss‚0.1004693452061963 /loss/FBt2/detection_loss‚0.5685510150023869 epoch‚3 _step‚130 /loss/FRt1/assignment_loss‚2.872001321440519 &loss/total_loss‚11.989392376854644 1loss/FBt2/assignment_loss‚0.26564313943422996 /loss/FRt1/detection_loss‚0.6280118773998639‚Þ]àbÒ‚(:× „¨åÒȘ›B! disk.dm-0.out‚16325.591796875 disk.rbd0.out‚0% disk.nvme0c0n1.out‚12289.02734375 disk.nvme0n1.in‚ 2.91796875& disk.nvme1c1n1.out‚4490.0361328125$ disk.nvme1n1.out‚4490.0361328125" proc.memory.rssMB‚ 2928.8203125 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.6640625) proc.memory.availableMB‚ 1440.16796875* proc.memory.percent‚17.876100540161133 network.recv‚326263 disk.nvme0c0n1.in‚ 2.91796875$ memory_percent‚91.20991230010986 cpu‚25.753050452642494 proc.cpu.threads‚16 network.sent‚ 2.238503e+06( disk./.usagePercent‚73.8345222119069% disk./.usageGB‚1133.5445022583008 disk.dm-0.in‚ 4.58203125 disk.dm-1.in‚0 disk.dm-1.out‚395.2724609375# disk.nvme0n1.out‚12289.02734375 disk.nvme1c1n1.in‚ 1.6640625 disk.md0.out‚ 26.9921875‡ÎµœÓ‚(:‘ „¨åÒíܺF gpu.0.gpu‚62.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.065' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.02166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûdD2Ô- _wandbruntime‚1052  _runtime‚1052cÆp\¦Õ÷{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10734646767377853 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7146885991096497 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5472332239151001 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904625415802002 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_1_of_1‚0.27454209327697754 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7040057182312012 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7187927961349487 *loss/val_total_loss‚12.702424907675804 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1339496523141861 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7731150388717651 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8802531361579895 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6883912086486816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7695389986038208 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.43417686223983765 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11611951887607574 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13968604803085327 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6016091704368591 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4941634237766266 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9737083911895752 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945785999298096 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14683279395103455 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5430306196212769 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.985230565071106 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9914430975914001 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7642478942871094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44683438539505005 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5939046144485474 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13235928118228912 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7003191709518433 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7153792977333069 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49138981103897095 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11972884088754654 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2497686892747879 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5643731355667114 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10067503899335861 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12105058133602142 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13042600452899933 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12086235731840134 7"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.57246333360672 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6059736609458923 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.123810ÙÖW/39559841156 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3486093580722809 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037036418914795 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9673323631286621 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1862475425004959 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010462312027812004 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8271777629852295 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9546107053756714 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.784239110769882e+09 _step‚131 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7146885991096497 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.975083589553833 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10825005173683167 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020244920626282692 +jet/accuracy_2_of_2‚0.11487839370965958 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4739229083061218 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.13477367162704468 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945785999298096 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9681769609451294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037036418914795 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7545138597488403 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3012891411781311 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3274807333946228 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02719305455684662 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7198008298873901 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10915182530879974 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37037035822868347 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637713432312 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11819186806678772 /particle/accuracy_4_of_4‚0.9408116936683655 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6393406987190247 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13076643645763397 jet/accuracy_4_of_5‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.983866810798645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0070679266937077045 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9619672894477844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10969239473342896 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6399630308151245 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 &particle/f_score‚0.602142870426178 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7592592835426331 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1911085695028305 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5851357579231262 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949694871902466 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5664961934089661 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5296251177787781 trainer/global_step‚1599 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12082219123840332 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9346648454666138 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7644935846328735 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11760867387056351 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13732603192329407 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841532111167908 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6142199039459229 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.59190434217453 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4661412537097931 *jet/accuracy_1_of_4‚0.9360715746879578 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.162234365940094 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13343749940395355 ,jet/accuracy_3_of_3‚0.047350864857435226 *jet/accuracy_2_of_4‚0.7105948328971863 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5310263633728027 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30987051129341125 *jet/accuracy_3_of_4‚0.0729844868183136 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7637944221496582 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12105058133602142 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17944321036338806 +particle/specificity‚0.9080939292907715 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9890509843826294 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1330908238887787 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.343963086605072 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09226803481578827 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.020303692668676376 +jet/accuracy_2_of_3‚0.45627284049987793 7validation_average_jet_accuracy‚0.32188138365745544 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6228410005569458 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13833333551883698 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 + Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6817528605461121 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13387976586818695 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9829583168029785 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5462921857833862 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8969687223434448 particle/accuracy_2_of_3‚1 _runtime‚1052.634241227 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268519401550293 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5939046144485474 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44755780696868896 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6437863111495972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003655052976682782 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3454648554325104 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10807670652866364 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5572916269302368 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5102040767669678 /particle/accuracy_3_of_3‚0.9995090961456299 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.826982319355011 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5755457282066345 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9930938482284546 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9822717308998108 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6358975172042847 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13835041224956512 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8366815447807312 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9886757135391235 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9865592122077942 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.37037035822868347 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5828929543495178 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5841314792633057 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2127990424633026 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5941959619522095 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10915182530879974 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6308230757713318 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6422052979469299 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9953219294548035 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6850441694259644 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 epoch‚3 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.860031008720398 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 /particle/accuracy_3_of_4‚0.9942177534103394 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6337820291519165 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7147265672683716‘>Öü{ 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9930938482284546 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9822717308998108 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6358975172042847 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13835041224956512 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8366815447807312 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9886757135391235 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9865592122077942 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.37037035822868347 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5828929543495178 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5841314792633057 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2127990424633026 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5941959619522095 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10915182530879974 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6308230757713318 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6422052979469299 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9953219294548035 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6850441694259644 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 epoch‚3 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.860031008720398 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 /particle/accuracy_3_of_4‚0.9942177534103394 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6337820291519165 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7147265672683716 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10734646767377853 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7146885991096497 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5472332239151001 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904625415802002 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_1_of_1‚0.27454209327697754 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7040057182312012 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7187927961349487 *loss/val_total_loss‚12.702424907675804 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1339496523141861 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7731150388717651 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8802531361579895 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6883912086486816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7695389986038208 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.43417686223983765 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11611951887607574 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13968604803085327 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6016091704368591 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4941634237766266 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9737083911895752 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945785999298096 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14683279395103455 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5430306196212769 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.985230565071106 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9914430975914001 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7642478942871094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44683438539505005 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5939046144485474 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13235928118228912 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7003191709518433 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7153792977333069 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49138981103897095 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11972884088754654 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2497686892747879 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5643731355667114 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10067503899335861 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12105058133602142 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13042600452899933 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12086235731840134 7"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.57246333360672 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6059736609458923 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12381039559841156 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3486093580722809 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037036418914795 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9673323631286621 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1862475425004959 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010462312027812004 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8271777629852295 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9546107053756714 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.784239110769882e+09 _step‚131 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7146885991096497 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.975083589553833 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10825005173683167 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020244920626282692 +jet/accuracy_2_of_2‚0.11487839370965958 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4739229083061218 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.13477367162704468 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945785999298096 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9681769609451294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037036418914795 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7545138597488403 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3012891411781311 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3274807333946228 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02719305455684662 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7198008298873901 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10915182530879974 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37037035822868347 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637713432312 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11819186806678772 /particle/accuracy_4_of_4‚0.9408116936683655 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6393406987190247 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13076643645763397 jet/accuracy_4_of_5‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.983866810798645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0070679266937077045 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9619672894477844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10969239473342896 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6399630308151245 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 &particle/f_score‚0.602142870426178 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7592592835426331 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1911085695028305 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5851357579231262 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949694871902466 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5664961934089661 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5296251177787781 trainer/global_step‚1599 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12082219123840332 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9346648454666138 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7644935846328735 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11760867387056351 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13732603192329407 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841532111167908 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6142199039459229 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.59190434217453 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4661412537097931 *jet/accuracy_1_of_4‚0.9360715746879578 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.162234365940094 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13343749940395355 ,jet/accuracy_3_of_3‚0.047350864857435226 *jet/accuracy_2_of_4‚0.7105948328971863 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5310263633728027 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30987051129341125 *jet/accuracy_3_of_4‚0.0729844868183136 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7637944221496582 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12105058133602142 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17944321036338806 +particle/specificity‚0.9080939292907715 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9890509843826294 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1330908238887787 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.343963086605072 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09226803481578827 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.020303692668676376 +jet/accuracy_2_of_3‚0.45627284049987793 7validation_average_jet_accuracy‚0.32188138365745544 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6228410005569458 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13833333551883698 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 + Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6817528605461121 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13387976586818695 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9829583168029785 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5462921857833862 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8969687223434448 particle/accuracy_2_of_3‚1 _runtime‚1052.634241227 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268519401550293 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5939046144485474 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44755780696868896 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6437863111495972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003655052976682782 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3454648554325104 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10807670652866364 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5572916269302368 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5102040767669678 /particle/accuracy_3_of_3‚0.9995090961456299 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.826982319355011 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5755457282066345 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1ƒ¸5`}^ׂ(:Ó “¨åÒ¿Ý™B disk.nvme1n1.in‚ 1.6640625 disk.md0.out‚ 27.578125 cpu‚25.759733724159293# disk.nvme1n1.out‚4554.732421875 disk.rbd0.out‚0) proc.memory.percent‚17.90027618408203 proc.cpu.threads‚16 network.recv‚329862' disk.nvme0c0n1.out‚12294.3486328125 disk.md0.in‚0 disk.dm-0.in‚ 4.58203125 disk.dm-1.out‚459.2021484375 disk.rbd0.in‚0# memory_percent‚91.2318229675293' proc.memory.availableMB‚ 1436.578125) disk./.usagePercent‚73.83452494512103% disk./.usageGB‚1133.5445442199707 disk.nvme0c0n1.in‚ 2.91796875% disk.nvme0n1.out‚12294.3486328125 disk.nvme1c1n1.in‚ 1.6640625! disk.dm-0.out‚16330.427734375 disk.dm-1.in‚0 proc.memory.rssMB‚ 2932.78125 network.sent‚ 2.298994e+06 disk.nvme0n1.in‚ 2.91796875% disk.nvme1c1n1.out‚4554.732421875$5äWØ‚(:’ “¨åÒˆÿàA gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚203.185' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.72833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ËNžÙ‚(:“ ¢¨åÒ©¾ëA gpu.0.gpu‚56.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚174.085' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚58.028333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘U2cÚ‚(:Ø ¢¨åÒ†ûA disk.md0.out‚ 28.62109375# disk.dm-0.out‚20828.07568359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.15234375 proc.memory.rssMB‚ 2932.78125) disk./.usagePercent‚74.17218522477317# disk./.usageGB‚1138.72847366333$ disk.nvme0n1.out‚16669.755859375& disk.nvme1c1n1.out‚4735.3525390625) proc.memory.availableMB‚ 1436.38671875) proc.memory.percent‚17.90027618408203 cpu‚25.758239644064908 proc.cpu.threads‚16 network.sent‚ 2.355276e+06 disk.nvme1n1.in‚ 1.6640625$ disk.nvme1n1.out‚4735.3525390625 disk.md0.in‚0! disk.dm-1.out‚515.43017578125 disk.nvme0c0n1.in‚ 4.15234375 disk.nvme1c1n1.in‚ 1.6640625# memory_percent‚91.2329912185669 disk.dm-0.in‚ 5.81640625& disk.nvme0c0n1.out‚16669.755859375 network.recv‚333503ª¬öð2Û- _wandbruntime‚1083  _runtime‚1083Un JÆÜÀ & lr-AdamW/pg2‚0.0002343039170050476 trainer/global_step‚1649 % _timestamp‚1.7842391419451923e+09 _runtime‚ 1083.80914806 _step‚132 & lr-AdamW/pg1‚0.0002343039170050476Þû¿§ËÝÅ _runtime‚ 1083.80914806 _step‚132 & lr-AdamW/pg1‚0.0002343039170050476 & lr-AdamW/pg2‚0.0002343039170050476 trainer/global_step‚1649 % _timestamp‚1.7842391419451923e+09„65]/2Þ- _wandbruntime‚1083  _runtime‚1083¸!j]ißÓO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8776717 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129362994961920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128002067783168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1360927178752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12521518 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1361015838720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97287653965824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16601418 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/actÛg[p!ive.large_pool.allocated‚13079023 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3744980 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401683854848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128002067783168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126491655662480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127848850380352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079021 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129363100661248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3744801 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128002084822528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079021 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1083.809896831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784239141946069e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95886064220672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1401798618624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129362994961920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 trainer/global_step‚1649 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16591275 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16591275 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29680441 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12521701 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079023 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16601418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 _step‚133 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1361015838720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95885970110976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128002084822528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127848954206138 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1357194717872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129363100661248 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 114763776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97287862839296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126491672701840Ñ“³Þ'àØO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128002084822528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127848954206138 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1357194717872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129363100661248 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 114763776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97287862839296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126491672701840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8776717 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129362994961920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128002067783168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1360927178752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12521518 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1361015838720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97287653965824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16601418 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13079023 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3744980 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401683854848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128002067783168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126491655662480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127848850380352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079021 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129363100661248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3744801 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128002084822528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079021 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1083.809896831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784239141946069e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95886064220672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1401798618624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129362994961920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 trainer/global_step‚1649 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16591275 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16591275 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29680441 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12521701 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079023 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16601418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 _step‚133 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1361015838720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95885970110976…%ÈA™2á- _wandbruntime‚1084  _runtime‚1084Ê„Eä&âÞM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12528933 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1361801561600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1361801561600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127923924400344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1361712894464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127924028226242 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128077217449472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8781777 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126565963227232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95942201948672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 % _timestamp‚1.7842391423431602e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128077234488832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3746975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126565946187872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12528748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚181 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97344811186688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _runtime‚1084.207266623 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086650 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129439036050432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128077234488832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402609238016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402492384256 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128077217449472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚185 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3747156 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129438930343936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086652 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95942107838976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16601095 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1357978212472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 2726297604î^yããM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚181 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97344811186688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _runtime‚1084.207266623 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086650 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129439036050432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128077234488832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402609238016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402492384256 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128077217449472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚185 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3747156 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129438930343936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086652 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95942107838976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16601095 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1357978212472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12528933 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1361801561600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1361801561600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127923924400344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1361712894464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127924028226242 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128077217449472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8781777 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126565963227232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95942201948672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 % _timestamp‚1.7842391423431602e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128077234488832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3746975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126565946187872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12528748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4†¹QêF2ä- _wandbruntime‚1084  _runtime‚1084‚²“zåý 1loss/FBt2/assignment_loss‚0.10050478683301446 -loss/FBt1/detection_loss‚0.89748749116067 epoch‚4 0loss/FRt2/assignment_loss‚1.4270135730900846 /loss/FRt1/detection_loss‚0.5654676996105081 /loss/FRt2/detection_loss‚1.4838448472032646 _runtime‚1084.207815464 2loss/SRqqt1/assignment_loss‚0.8416545035493457 0loss/SRqqt2/detection_loss‚0.832228865651148 trainer/global_step‚1649 2loss/SRqqt2/assignment_loss‚0.7187778873271626 1loss/SRqqt1/detection_loss‚1.2001079398911032 /loss/FBt2/detection_loss‚1.9719786135302413 &loss/total_loss‚12.929205250393832 % _timestamp‚1.7842391423441129e+09 _step‚135 .loss/FRt1/assignment_loss‚2.82410643968199 1loss/FBt1/assignment_loss‚0.06603260286529933ì447æ‚ 1loss/SRqqt1/detection_loss‚1.2001079398911032 /loss/FBt2/detection_loss‚1.9719786135302413 &loss/total_loss‚12.929205250393832 % _timestamp‚1.7842391423441129e+09 _step‚135 .loss/FRt1/assignment_loss‚2.82410643968199 1loss/FBt1/assignment_loss‚0.06603260286529933 1loss/FBt2/assignment_loss‚0.10050478683301446 -loss/FBt1/detection_loss‚0.89748749116067 epoch‚4 0loss/FRt2/assignment_loss‚1.4270135730900846 /loss/FRt1/detection_loss‚0.5654676996105081 /loss/FRt2/detection_loss‚1.4838448472032646 _runtime‚1084.207815464 2loss/SRqqt1/assignment_loss‚0.8416545035493457 0loss/SRqqt2/detection_loss‚0.832228865651148 trainer/global_step‚1649 2loss/SRqqt2/assignment_loss‚0.7187778873271626‡ìišoç‚(:’ ±¨åÒçàöA gpu.0.gpu‚63.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.439' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.47966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J°Ã_[è‚(:Ð ±¨åÒý˜†B disk.nvme1n1.in‚ 1.6640625! disk.nvme1n1.out‚ 4996.3671875 network.sent‚ 2.361759e+06! disk.dm-1.out‚523.39111328125 disk.nvme1c1n1.in‚ 1.6640625$ memory_percent‚91.23227596282959) proc.memory.availableMB‚ 1436.50390625 proc.memory.rssMB‚ 2932.78125) proc.memory.percent‚17.90027618408203 network.recv‚337356$ disk./.usageGB‚1136.258644104004 disk.md0.out‚ 31.74609375 disk.dm-1.in‚0& disk.nvme0n1.out‚17489.60205078125 proc.cpu.threads‚16 disk.md0.in‚0 disk.dm-0.in‚ 5.8203125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17489.60205078125 cpu‚25.756093256119325) disk./.usagePercent‚74.01131047738178" disk.dm-0.out‚21894.6494140625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.15625 disk.nvme0n1.in‚4.15625# disk.nvme1c1n1.out‚ 4996.3671875ÿZõD`é‚(:Õ À¨åÒʨóA disk.dm-0.in‚ 5.8203125 disk.nvme0n1.in‚4.15625 disk.nvme1c1n1.in‚ 1.6640625 proc.cpu.threads‚16) disk./.usagePercent‚74.01132240413435 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚17536.5771484375 cpu‚25.754015516568522 network.sent‚ 2.424045e+06" disk.dm-0.out‚21943.5419921875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.15625% disk.nvme0n1.out‚17536.5771484375' disk.nvme1c1n1.out‚5000.15771484375 proc.memory.rssMB‚ 2932.78125) proc.memory.percent‚17.90027618408203 disk.nvme1n1.in‚ 1.6640625 disk.md0.in‚0! disk.dm-1.out‚523.98974609375$ memory_percent‚91.23430252075195' proc.memory.availableMB‚ 1436.171875 network.recv‚341135% disk./.usageGB‚1136.2588272094727% disk.nvme1n1.out‚5000.15771484375 disk.md0.out‚ 32.35546875³+Åê‚(:’ À¨åÒ­§B gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚222.106' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.03533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÉ’2ë- _wandbruntime‚1115  _runtime‚1115ùp9ÍÆìÀ & lr-AdamW/pg1‚0.0002341116026494726 & lr-AdamW/pg2‚0.0002341116026494726 trainer/global_step‚1699 $ _timestamp‚1.784239173531924e+09 _runtime‚1115.395891044 _step‚136tà-ËíÅ trainer/global_step‚1699 $ _timestamp‚1.784239173531924e+09 _runtime‚1115.395891044 _step‚136 & lr-AdamW/pg1‚0.0002341116026494726 & lr-AdamW/pg2‚0.0002341116026494726ˆFP¬M2î- _wandbruntime‚1115  _runtime‚1115…ØhÛ'ïÕO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100135189996544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131759568137728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1442446152192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1400277118976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133159828217344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17084127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 trainer/global_step‚1699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133159828217344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _step‚137 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12883095 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133159933916672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9029521 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100135396772864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131602719006546 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1400365778944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17094270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784239173532807e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3853754 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98692950620672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1396520035106 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130206198971440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13460473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130206181932080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131759551098368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112666624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1115.400348189 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3853578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460471 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883275 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9029517 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1442333485568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1400365778944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131602615180760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460471 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1400277118976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17094270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1396433248680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17084127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30554743 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98692856510976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131759551098368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17084127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 trainer/global_step‚1699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133159828217344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _step‚137 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12883095 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133159933916672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9029521 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100135396772864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131602719006546 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1400365778944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17094270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784239173532807e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3853754 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98692950620672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1396520035106 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130206198971440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13460473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130206181932080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131759551098368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112666624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1115.400348189 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3853578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460471 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883275 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9029517 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1442333485568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1400365778944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131602615180760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460471 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1400277118976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17094270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1396433248680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760‰Þ žn2ñ- _wandbruntime‚1115  _runtime‚1115hc{Ž òàM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131834717804032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131677793026650 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3855696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_byt’û\¢Ú es.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17104104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468100 _runtime‚1115.792107746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17093947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30572206 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131834700764672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚1699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130280472457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚138 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1401062834688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12890269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131677689200752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3855873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1443260990464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9034573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98748994238976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468100 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1397303529818 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401062834688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130280472457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚138 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1401062834688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12890269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131677689200752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3855873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1443260990464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9034573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98748994238976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468100 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1397303529818 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401062834688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3855696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17104104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468100 _runtime‚1115.792107746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17093947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ŠB¹5±2ô- _wandbruntime‚1115  _runtime‚1115”¹jõÿ /loss/FRt2/assignment_loss‚1.420474737482583 2loss/SRqqt2/assignment_loss‚0.6868877118849875 /loss/FBt1/detection_loss‚0.8436985212524537 trainer/global_step‚1699 _step‚139 2loss/SRqqt1/assignment_loss‚0.8398102160760512 1loss/FBt1/assignment_loss‚0.09185104984606354 /loss/FRt1/detection_loss‚0.6448564075105813 /loss/FRt2/detection_loss‚1.3958006373400258 1loss/SRqqt1/detection_loss‚1.2254278399878078 0loss/FRt1/assignment_loss‚2.6505063911119366 1loss/SRqqt2/detection_loss‚0.7716680398077793 %loss/total_loss‚12.11416892012749 $ _timestamp‚1.784239173928951e+09 _runtime‚1115.792648457 1loss/FBt2/assignment_loss‚0.09227199790359464 /loss/FBt2/detection_loss‚1.4509153699236257 epoch‚4Ø2Ý0 ö„ /loss/FRt2/assignment_loss‚1.420474737482583 2loss/SRqqt2/assignment_loss‚0.6868877118849875 /loss/FBt1/detection_loss‚0.8436985212524537 trainer/global_step‚1699 _step‚139 2loss/SRqqt1/assignment_loss‚0.8398102160760512 1loss/FBt1/assignment_loss‚0.09185104984606354 /loss/FRt1/detection_loss‚0.6448564075105813 /loss/FRt2/detection_loss‚1.3958006373400258 1loss/SRqqt1/detection_loss‚1.2254278399878078 0loss/FRt1/assignment_loss‚2.6505063911119366 1loss/SRqqt2/detection_loss‚0.7716680398077793 %loss/total_loss‚12.11416892012749 $ _timestamp‚1.784239173928951e+09 _runtime‚1115.792648457 1loss/FBt2/assignment_loss‚0.09227199790359464 /loss/FBt2/detection_loss‚1.4509153699236257 epoch‚4‹ÃQ2[÷‚(:Ð ϨåÒ÷ú†B disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.90027618408203 disk.nvme1c1n1.in‚ 1.6640625 disk.nvme1n1.in‚ 1.6640625 disk.dm-0.in‚ 5.9140625 disk.rbd0.in‚0& proc.memory.availableMB‚ 1436.28125 network.sent‚ 2.428862e+06 network.recv‚344782) disk./.usagePercent‚73.80669212869316( disk.nvme0c0n1.out‚17544.54736328125' disk.nvme1c1n1.out‚5001.22021484375 disk.md0.in‚0$ memory_percent‚91.23363494873047 cpu‚25.752417602154242% disk./.usageGB‚1133.1172409057617 disk.nvme0c0n1.in‚4.25& disk.nvme0n1.out‚17544.54736328125% disk.nvme1n1.out‚5001.22021484375 disk.md0.out‚ 32.7890625! disk.dm-1.out‚524.25927734375 proc.memory.rssMB‚ 2932.78125 proc.cpu.threads‚16 disk.nvme0n1.in‚4.25# disk.dm-0.out‚21951.35986328125HwX‹’ø‚(:‡ ϨåÒˆí·B gpu.0.gpu‚64.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.198' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨o=6ù‚(:’ Þ¨åÒö½íA gpu.0.gpu‚92.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.629' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.20966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íõl¿Yú‚(:Î Þ¨åÒÿÑúA$ memory_percent‚91.23406410217285) proc.memory.percent‚17.90027618408203% disk.nvme0n1.out‚17552.5888671875 network.sent‚ 2.491091e+06 network.recv‚348973% disk./.usageGB‚1133.1800422668457 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.25 disk.nvme0n1.in‚4.25 disk.nvme1n1.in‚ 1.6640625) disk./.usagePercent‚73.81078275634884 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚5002.64990234375% disk.nvme1n1.out‚5002.64990234375 disk.md0.in‚0 proc.memory.rssMB‚ 2932.78125 cpu‚25.75043931327285 disk.dm-1.out‚ 524.365234375' disk.nvme0c0n1.out‚17552.5888671875 disk.md0.out‚ 33.11328125 disk.dm-0.in‚ 5.9140625( proc.memory.availableMB‚ 1436.2109375 proc.cpu.threads‚16 disk.nvme1c1n1.in‚ 1.6640625# disk.dm-0.out‚21960.02294921875·zLÌ2û- _wandbruntime‚1146  _runtime‚1146k]ÅòÉüà _runtime‚1146.585517058 _step‚140 ' lr-AdamW/pg1‚0.00023391185209334684 ' lr-AdamW/pg2‚0.00023391185209334684 trainer/global_step‚1749 % _timestamp‚1.7842392047215447e+09njSÎýÈ _step‚140 ' lr-AdamW/pg1‚0.00023391185209334684 ' lr-AdamW/pg2‚0.00023391185209334684 trainer/global_step‚1749 % _timestamp‚1.7842392047215447e+09 _runtime‚1146.585517058Œ+ãç2þ- _wandbruntime‚1146  _runtime‚1146®nLTº%ÿØO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17576979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17587122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚198 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13841921 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13244723 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135356379981168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1435758565914 trainer/global_step‚1749 _runtime‚1146.589365694 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1439715719168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1482957843968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133920708201680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1435671779488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9282317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚194 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101499742910976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102982918017024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17587122 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1439715719168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13841923 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136956661472768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135356483806954 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 % _timestamp‚1.7842392047224283e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135517051452928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136956767172096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136956661472768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13841923 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135517034413568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135517051452928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31429045 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3962402 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101499837020672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.alœ3‹i$located‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760µ¶ã'€ ÝO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 % _timestamp‚1.7842392047224283e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135517051452928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136956767172096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136956661472768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13841923 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135517034413568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135517051452928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31429045 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3962402 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101499837020672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17576979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17587122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚198 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13841921 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13244723 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135356379981168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1435758565914 trainer/global_step‚1749 _runtime‚1146.589365694 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1439715719168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1482957843968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133920708201680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1435671779488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9282317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚194 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101499742910976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102982918017024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17587122 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1439715719168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13841923 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136956661472768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135356483806954 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ãºè2 - _wandbruntime‚1146  _runtime‚1146°ç&‚ áM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135592184079872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135431557827058 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚142 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚ 1146.98225248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9287373 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103039645439488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17586799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 trainer/global_step‚1749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1440412774912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135431454001160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1436455274088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135592184079872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101555974748672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137032702561280 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137032596854784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1483896334336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135592201119232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9287377 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133995015766432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7842392051181161e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13251779 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17596956 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135592201119232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3964601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3964406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13849552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17596956 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1436542060626 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13849550 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17586799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚195 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157N‹/ì&ƒ æM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1440501442048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13251779 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17596956 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135592201119232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3964601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3964406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13849552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17596956 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1436542060626 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13849550 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17586799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚195 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135592184079872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135431557827058 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚142 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚ 1146.98225248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9287373 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103039645439488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17586799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 trainer/global_step‚1749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1440412774912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135431454001160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1436455274088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135592184079872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101555974748672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137032702561280 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137032596854784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1483896334336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135592201119232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9287377 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133995015766432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7842392051181161e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9535117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140753494728192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105830190024192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1478976999424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1523560713216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140753494728192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139274517728768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9535121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18079974 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 $ _timestamp‚1.784239236326757e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4071266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140753600427520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139274534768128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18069831 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139274517728768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071092 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104306723420672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1479065659392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14223373 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18069831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105830409383424«±Ü-â' ÜO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1474997096722 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140753494728192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105830190024192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1478976999424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1523560713216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140753494728192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139274517728768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9535121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18079974 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 $ _timestamp‚1.784239236326757e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4071266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140753600427520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139274534768128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18069831 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139274517728768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071092 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104306723420672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1479065659392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14223373 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18069831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105830409383424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1479065659392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚178 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140753600427520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9535117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18079651 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140829535816704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1475693804896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1524373854720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140829535816704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104362767038976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231002 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1479851382272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137709524996672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32310651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13613376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105887140893696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18079651 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139349684434432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9540173 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18089808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1479762715136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140829430110208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139349684434432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1479851382272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137709542036032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18089808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231002 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479762715136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1524490708480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 _runtime‚1178.587242319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139185322627466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139185218801568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0#M£_í&“ çM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1479762715136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140829430110208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139349684434432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1479851382272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137709542036032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18089808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231002 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479762715136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1524490708480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 _runtime‚1178.587242319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139185322627466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139185218801568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18079651 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140829535816704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1475693804896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1524373854720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140829535816704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104362767038976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231002 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1479851382272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137709524996672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32310651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13613376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105887140893696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18079651 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139349684434432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9540173 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18089808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157’BHa2” - _wandbruntime‚1178  _runtime‚1178BàF • ý 2loss/SRqqt1/assignment_loss‚0.7899016673654407 1loss/FBt1/assignment_loss‚0.11419003540694024 &loss/total_loss‚12.467076367274602 epoch‚4 _runtime‚1178.587782781 /loss/FRt2/assignment_loss‚1.464674248546362 .loss/FRt2/detection_loss‚1.375470695039829 1loss/SRqqt1/detection_loss‚1.1094471906864667 trainer/global_step‚1799 % _timestamp‚1.7842392367240992e+09 _step‚147 2loss/SRqqt2/assignment_loss‚0.7591203511195493 0loss/FBt2/assignment_loss‚0.0751526169757426 /loss/FRt1/detection_loss‚0.5180049743720451 1loss/SRqqt2/detection_loss‚0.7906544035487904 /loss/FBt1/detection_loss‚1.0409976602200361 .loss/FRt1/assignment_loss‚2.75155850285298 /loss/FBt2/detection_loss‚1.6779040211404208}úƒ– ‚ 2loss/SRqqt1/assignment_loss‚0.7899016673654407 1loss/FBt1/assignment_loss‚0.11419003540694024 &loss/total_loss‚12.467076367274602 epoch‚4 _runtime‚1178.587782781 /loss/FRt2/assignment_loss‚1.464674248546362 .loss/FRt2/detection_loss‚1.375470695039829 1loss/SRqqt1/detection_loss‚1.1094471906864667 trainer/global_step‚1799 % _timestamp‚1.7842392367240992e+09 _step‚147 2loss/SRqqt2/assignment_loss‚0.7591203511195493 0loss/FBt2/assignment_loss‚0.0751526169757426 /loss/FRt1/detection_loss‚0.5180049743720451 1loss/SRqqt2/detection_loss‚0.7906544035487904 /loss/FBt1/detection_loss‚1.0409976602200361 .loss/FRt1/assignment_loss‚2.75155850285298 /loss/FBt2/detection_loss‚1.6779040211404208“8Mî§]— ‚(:Ò ‹©åÒ˜µÔA disk.dm-1.in‚0 network.sent‚ 2.56386e+06 disk.nvme1c1n1.in‚ 1.86328125 disk.md0.in‚0" disk.nvme0c0n1.in‚ 105.02734375 network.recv‚359997 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21665.01318359375) proc.memory.percent‚17.90027618408203 cpu‚25.74501075483969 proc.cpu.threads‚16 disk.dm-1.out‚524.9384765625) disk./.usagePercent‚74.17662073435719% disk./.usageGB‚1138.7965698242188" disk.nvme1c1n1.out‚ 5049.546875 disk.nvme1n1.in‚ 1.86328125 disk.md0.out‚ 34.87109375 disk.rbd0.in‚0& disk.nvme0n1.out‚21665.01318359375 disk.nvme1n1.out‚ 5049.546875 disk.dm-0.in‚ 106.890625# disk.dm-0.out‚26115.05322265625 disk.nvme0n1.in‚ 105.02734375# memory_percent‚91.2384033203125" proc.memory.availableMB‚1435.5 proc.memory.rssMB‚ 2932.78125P³Cßž˜ ‚(:“ ‹©åÒÜ´ÊA gpu.0.gpu‚49.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.689' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.562999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍzO~ž™ ‚(:“ š©åÒù¹êA gpu.0.gpu‚48.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚116.519' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.839666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L#YNaš ‚(:Ö š©åÒõÄúA cpu‚25.74341047195244 proc.cpu.threads‚16 disk.nvme0n1.in‚ 106.484375 disk.nvme1n1.in‚ 2.73828125 disk.md0.out‚ 35.171875 disk.dm-0.in‚ 109.22265625# disk.dm-0.out‚28038.06494140625 disk.dm-1.in‚0) disk./.usagePercent‚74.21145927554679$ disk./.usageGB‚1139.331428527832 disk.nvme0c0n1.in‚ 106.484375( disk.nvme0c0n1.out‚23153.64892578125 disk.rbd0.out‚0 network.recv‚364040& disk.nvme0n1.out‚23153.64892578125 disk.nvme1c1n1.in‚ 2.73828125$ disk.nvme1c1n1.out‚ 5484.86328125 disk.md0.in‚0 disk.dm-1.out‚ 525.216796875 disk.rbd0.in‚0) proc.memory.percent‚17.90027618408203 network.sent‚ 2.626293e+06" disk.nvme1n1.out‚ 5484.86328125$ memory_percent‚91.23783111572266& proc.memory.availableMB‚ 1435.59375 proc.memory.rssMB‚ 2932.78125ñìµ2› - _wandbruntime‚1209  _runtime‚1209Xó«Çœ Á ' lr-AdamW/pg1‚0.00023349009417465228 ' lr-AdamW/pg2‚0.00023349009417465228 trainer/global_step‚1849 # _timestamp‚1.78423926770299e+09 _runtime‚1209.566965828 _step‚148i˜ÆUÌ Æ _runtime‚1209.566965828 _step‚148 ' lr-AdamW/pg1‚0.00023349009417465228 ' lr-AdamW/pg2‚0.00023349009417465228 trainer/global_step‚1849 # _timestamp‚1.78423926770299e+09”mu¿j2ž - _wandbruntime‚1209  _runtime‚1209˜4cÜ'Ÿ ÖO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108677555762176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144550433682944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14604823 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33177649 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 % _timestamp‚1.7842392677038758e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107113609820672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1518326939648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144550327983616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 trainer/global_step‚1849 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18562683 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9787921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1564171592192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4177873 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141349777780240 _runtime‚ 1209.5713304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143032001043968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143032001043968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14604823 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144550327983616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚193 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13965790 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1514148841104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1514235627530 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13965987 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9787917 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1518326939648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142863909581984 _step‚149 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141349760740880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1518415599616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13965790 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1514148841104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1514235627530 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13965987 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9787917 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1518326939648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142863909581984 _step‚149 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141349760740880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1518415599616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144550327983616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚193 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144626369072128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14612450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18582660 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143107150710272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚150 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4180059 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1519112655360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18572503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4179890 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18582660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1514932335704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144626263365632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108734716186112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9792973 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142938983601976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143107150710272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1519201322496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107169747548672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142939087427874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1564853880832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141424068305632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14612450 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143107167749632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13972863 _runtime‚1209.963510567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1519201322496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108734507319808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107169653438976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7842392680993757e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1564968637440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14612452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14612452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18572503 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1519112655360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9792977 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1515019122242 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0´ÿí&£ çM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142938983601976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143107150710272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1519201322496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107169747548672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142939087427874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1564853880832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141424068305632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14612450 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143107167749632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13972863 _runtime‚1209.963510567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1519201322496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108734507319808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107169653438976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7842392680993757e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1564968637440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14612452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14612452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18572503 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1519112655360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9792977 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1515019122242 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141424051266272 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144626369072128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14612450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18582660 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143107150710272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚150 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4180059 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1519112655360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18572503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4179890 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18582660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1514932335704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144626263365632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108734716186112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9792973–Ò"(|2¤ - _wandbruntime‚1209  _runtime‚1209i¤íµ¥ ÿ /loss/FRt1/assignment_loss‚2.775000806850723 1loss/FBt1/assignment_loss‚0.08685540661514755 /loss/FRt2/detection_loss‚1.4626255519461766 1loss/SRqqt1/detection_loss‚1.2732477758634597 _step‚151 /loss/FRt2/assignment_loss‚1.306584446660934 0loss/FBt2/assignment_loss‚0.2597101397566048 /loss/FRt1/detection_loss‚0.5669998266780802 &loss/total_loss‚12.110603561029182 1loss/SRqqt2/detection_loss‚0.7102669676716934 /loss/FBt1/detection_loss‚0.9686646001760515 epoch‚4 trainer/global_step‚1849 % _timestamp‚1.7842392681003342e+09 _runtime‚1209.963971897 2loss/SRqqt1/assignment_loss‚0.7652898352117614 2loss/SRqqt2/assignment_loss‚0.7388193516561583 /loss/FBt2/detection_loss‚1.1965388519423914†Þu ¦ „ &loss/total_loss‚12.110603561029182 1loss/SRqqt2/detection_loss‚0.7102669676716934 /loss/FBt1/detection_loss‚0.9686646001760515 epoch‚4 trainer/global_step‚1849 % _timestamp‚1.7842392681003342e+09 _runtime‚1209.963971897 2loss/SRqqt1/assignment_loss‚0.7652898352117614 2loss/SRqqt2/assignment_loss‚0.7388193516561583 /loss/FBt2/detection_loss‚1.1965388519423914 /loss/FRt1/assignment_loss‚2.775000806850723 1loss/FBt1/assignment_loss‚0.08685540661514755 /loss/FRt2/detection_loss‚1.4626255519461766 1loss/SRqqt1/detection_loss‚1.2732477758634597 _step‚151 /loss/FRt2/assignment_loss‚1.306584446660934 0loss/FBt2/assignment_loss‚0.2597101397566048 /loss/FRt1/detection_loss‚0.5669998266780802—öÊA£\§ ‚(:Ñ ©©åÒë¾ùA network.recv‚367741) disk./.usagePercent‚74.22627279918021 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚5489.6142578125" disk.dm-0.out‚28095.1064453125 disk.dm-1.in‚0$ memory_percent‚91.63017272949219 disk.nvme0n1.in‚ 109.09765625 disk.nvme1c1n1.in‚ 2.73828125 disk.dm-1.out‚ 525.345703125 proc.cpu.threads‚16# disk.nvme0c0n1.out‚ 23206.703125 disk.md0.out‚ 35.453125% proc.memory.availableMB‚ 1371.3125 proc.memory.rssMB‚ 2932.78125) proc.memory.percent‚17.90027618408203 cpu‚25.7418627494481 network.sent‚ 2.631143e+06$ disk./.usageGB‚1139.558853149414" disk.nvme0c0n1.in‚ 109.09765625! disk.nvme0n1.out‚ 23206.703125& disk.nvme1c1n1.out‚5489.6142578125 disk.nvme1n1.in‚ 2.73828125 disk.md0.in‚0 disk.dm-0.in‚ 111.83984375‰Ž?3¨ ‚(:’ ©©åÒîˬB gpu.0.gpu‚16.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚175.667' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚58.555666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÚt¹m© ‚(:â ¸©åÒÈæîA network.sent‚ 2.692905e+06 network.recv‚371538) disk./.usagePercent‚74.22466219063594 disk.dm-0.in‚ 121.16796875! disk.nvme0c0n1.in‚ 118.4296875( disk.nvme0c0n1.out‚23650.63916015625' disk.nvme1c1n1.out‚5680.83251953125 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚23650.63916015625 disk.nvme1c1n1.in‚ 2.73828125 disk.nvme1n1.in‚ 2.73828125 proc.memory.rssMB‚ 2932.78125) proc.memory.percent‚17.90027618408203% disk./.usageGB‚1139.5341262817383# disk.dm-0.out‚28729.73486328125! disk.dm-1.out‚525.46728515625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 118.4296875% disk.nvme1n1.out‚5680.83251953125 disk.md0.out‚ 35.6328125$ memory_percent‚91.63036346435547& proc.memory.availableMB‚ 1371.28125 cpu‚25.74055681328507 proc.cpu.threads‚16 †Ú“’ª ‚(:‡ ¸©åÒÂƦB gpu.0.gpu‚66.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚191.487' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚63.829 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00"2« - _wandbruntime‚1241  _runtime‚1241v¹|=ɬ à _runtime‚1241.180456913 _step‚152 ' lr-AdamW/pg1‚0.00023326811390080645 ' lr-AdamW/pg2‚0.00023326811390080645 trainer/global_step‚1899 % _timestamp‚1.7842392993164935e+09!UÀïέ È % _timestamp‚1.7842392993164935e+09 _runtime‚1241.180456913 _step‚152 ' lr-AdamW/pg1‚0.00023326811390080645 ' lr-AdamW/pg2‚0.00023326811390080645 trainer/global_step‚1899˜”ˆ›ý2® - _wandbruntime‚1241  _runtime‚1241>JGT¯ ÙO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14986271 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146789501398528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109920496220672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148347161239040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148347266938368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1553387371912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14986273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19055535 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14325209 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10040721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1557765539840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1557765539840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4284659 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145064287010480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4284492 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111525176754688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1604555284480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1604680534016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146617674382392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚1899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19065678 _runtime‚1241.182325344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146789484359168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148347161239040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146789484359168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146617778208178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1557676879872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14325380 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0À3ä'° ÞO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148347161239040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148347266938368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1553387371912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14986273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19055535 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14325209 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10040721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1557765539840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1557765539840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4284659 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145064287010480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4284492 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111525176754688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1604555284480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1604680534016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146617674382392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚1899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19065678 _runtime‚1241.182325344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146789484359168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148347161239040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146789484359168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146617778208178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1557676879872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14325380 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14986271 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146789501398528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109920496220672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14332379 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚1241.576180984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19075512 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14993902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109976633948672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148423096621056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146864651064832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1558551262720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148423202327552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1558462595584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146692852228282 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145138577535872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145138594575232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842392997120748e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1554257653050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1558551262720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1605363191296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111581903030272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19065355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4286606 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1605496822272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148423202327552 _step‚154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4286780 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486r†4#R³ éM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14993902 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146692748402384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1554170866512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14993902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109976633948672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148423096621056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146864651064832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1558551262720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148423202327552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1558462595584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146692852228282 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145138577535872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145138594575232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842392997120748e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1554257653050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1558551262720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1605363191296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111581903030272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19065355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4286606 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1605496822272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148423202327552 _step‚154 N>DeviceStat•ÆŸH sMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4286780 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1558462595584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111582130770944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14993900 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14332379 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚1241.576180984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19075512 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152143994494464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150371439182800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1597115480064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1592625902720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1645214977024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597026820096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150546984713728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150546984713728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112727382620672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152143994494464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15367723 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597026820096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1597115480064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19548387 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1592712689146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19558530 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19558530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4390044 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4389867 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15367721 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10293517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112727288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1645083436032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152144100193792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚1949 _runtime‚ 1272.95338265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150546967674368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150546967674368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15367723 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14683384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148778813280080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14683565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 Wã'À ÝO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597026820096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150546984713728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150546984713728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112727382620672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152143994494464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15367723 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597026820096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1597115480064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19548387 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1592712689146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19558530 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19558530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4390044 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4389867 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15367721 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10293517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112727288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1645083436032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152144100193792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚1949 _runtime‚ 1272.95338265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150546967674368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150546967674368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15367723 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14683384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148778813280080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14683565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 $ _timestamp‚1.784239331086067e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152143994494464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150371439182800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1597115480064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1592625902720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1645214977024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760}'‡×2Á - _wandbruntime‚1273  _runtime‚1273,WµÆ äM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10298577 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10298573 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4391842 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚1949 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19568364 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150446617028690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚158 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150622117340672 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148853103805472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15375350 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150622134380032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112783426238976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _runtime‚1273.347647441 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 EY$ain_batch_end/active.all.allocated‚34943716 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1597812535808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152220035582976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15375352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1646021033472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15375350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148853120844832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1593409397320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15375352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4392018 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14690415 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1597812535808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645893693952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150622134380032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114429541382144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7842393314834747e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19558207 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19558207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150622117340672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34933557 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1597901202944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150446513202792`r3ï&à éM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚1949 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19568364 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150446617028690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚158 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150622117340672 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148853103805472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15375350 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150622134380032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112783426238976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _runtime‚1273.347647441 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152220035582976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15375352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1646021033472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15375350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148853120844832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1593409397320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15375352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4392018 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14690415 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1597812535808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645893693952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150622134380032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114429541382144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7842393314834747e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19558207 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19558207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150622117340672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34933557 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1597901202944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150446513202792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10298577 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10298573 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4391842 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812žá¾,2Ä - _wandbruntime‚1273  _runtime‚1273›KøcÅ ý /loss/FRt2/detection_loss‚1.4191875840051595 0loss/SRqqt1/detection_loss‚1.010984221282993 _runtime‚1273.348512992 1loss/SRqqt2/assignment_loss‚0.707127854900792 /loss/FBt1/detection_loss‚0.6231222139071991 epoch‚4 .loss/FRt2/assignment_loss‚1.42398274989802 1loss/SRqqt2/detection_loss‚0.7441401231419704 trainer/global_step‚1949 1loss/FBt1/assignment_loss‚0.11172295870847926 .loss/FBt2/detection_loss‚1.471555785221212 % _timestamp‚1.7842393314848905e+09 _step‚159 &loss/total_loss‚11.787058241921788 0loss/FRt1/assignment_loss‚2.5707089600636976 2loss/SRqqt1/assignment_loss‚0.7328101643964472 1loss/FBt2/assignment_loss‚0.35447703601205244 /loss/FRt1/detection_loss‚0.61723859038376459ŸÜ‘Æ ‚ /loss/FBt1/detection_loss‚0.6231222139071991 epoch‚4 .loss/FRt2/assignment_loss‚1.42398274989802 1loss/SRqqt2/detection_loss‚0.7441401231419704 trainer/global_step‚1949 1loss/FBt1/assignment_loss‚0.11172295870847926 .loss/FBt2/detection_loss‚1.471555785221212 % _timestamp‚1.7842393314848905e+09 _step‚159 &loss/total_loss‚11.787058241921788 0loss/FRt1/assignment_loss‚2.5707089600636976 2loss/SRqqt1/assignment_loss‚0.7328101643964472 1loss/FBt2/assignment_loss‚0.35447703601205244 /loss/FRt1/detection_loss‚0.6172385903837645 /loss/FRt2/detection_loss‚1.4191875840051595 0loss/SRqqt1/detection_loss‚1.010984221282993 _runtime‚1273.348512992 1loss/SRqqt2/assignment_loss‚0.707127854900792Ÿà u3Ç ‚(:’ å©åÒªƒÿA gpu.0.gpu‚79.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.574' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.52466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨±9qÈ ‚(:æ å©åÒ±š˜B disk.md0.out‚ 37.13671875 disk.dm-0.in‚ 121.30859375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 118.5703125( proc.memory.availableMB‚ 1371.1484375& disk.nvme0n1.out‚23686.85302734375% disk.nvme1n1.out‚5712.49072265625# disk.dm-0.out‚28794.13916015625! disk.dm-1.out‚525.72998046875$ memory_percent‚91.63117408752441 proc.memory.rssMB‚ 2932.78125) disk./.usagePercent‚74.23140527836722% disk./.usageGB‚1139.6376495361328 disk.md0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚23686.85302734375 network.recv‚382501 disk.nvme1c1n1.in‚ 2.73828125' disk.nvme1c1n1.out‚5712.49072265625 disk.rbd0.in‚0) proc.memory.percent‚17.90027618408203 cpu‚25.736214886765577 proc.cpu.threads‚16 network.sent‚ 2.765534e+06 disk.nvme0n1.in‚ 118.5703125 disk.nvme1n1.in‚ 2.73828125Ø·¬ÏÉ ‚(:’ ô©åÒ¹²×A gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚219.989' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.32966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÎlànÊ ‚(:ã ô©åÒ¯ŽèA disk.dm-1.in‚0! disk.dm-1.out‚525.87353515625" disk.nvme0c0n1.in‚ 118.58203125( proc.memory.availableMB‚ 1370.9609375# disk.dm-0.out‚31571.31689453125 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2932.78125 network.sent‚ 2.82676e+06 disk.nvme1c1n1.in‚ 2.73828125& disk.nvme0n1.out‚26362.42041015625# memory_percent‚91.6323184967041) proc.memory.percent‚17.90027618408203 cpu‚25.73466240807428 proc.cpu.threads‚16 network.recv‚386398) disk./.usagePercent‚74.45352464560052$ disk./.usageGB‚1143.047737121582% disk.nvme1n1.out‚5814.85205078125 disk.dm-0.in‚ 121.3203125( disk.nvme0c0n1.out‚26362.42041015625 disk.nvme0n1.in‚ 118.58203125' disk.nvme1c1n1.out‚5814.85205078125 disk.nvme1n1.in‚ 2.73828125 disk.md0.in‚0 disk.md0.out‚ 37.41015625\?Íü2Ë - _wandbruntime‚1304  _runtime‚1304ÿžÞÄÌ ¾ % lr-AdamW/pg1‚0.000232802022010093 % lr-AdamW/pg2‚0.000232802022010093 trainer/global_step‚1999 $ _timestamp‚1.784239362487666e+09 _runtime‚1304.351648326 _step‚160Š'K€ÉÍ Ã _runtime‚1304.351648326 _step‚160 % lr-AdamW/pg1‚0.000232802022010093 % lr-AdamW/pg2‚0.000232802022010093 trainer/global_step‚1999 $ _timestamp‚1.784239362487666e+09 À¦f#2Î - _wandbruntime‚1304  _runtime‚1304°–,à'Ï ÚO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636376760320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1685560890368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4496661 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20051382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20041239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10546321 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154304468028928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20051382 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155940827749888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631864433528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636376760320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15749173 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636465420288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115534269020672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154304450989568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15749171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20041239 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155940933449216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155940827749888 % _timestamp‚1.7842393624886086e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15749173 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154304450989568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10546317 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152493339549680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154304468028928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15749171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚1304.352427397ñm"] Ð ßO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15042982 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15042803 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4496486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631951219954 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4496661 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20051382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20041239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10546321 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154304468028928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20051382 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155940827749888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.smaXí½ˆll_pool.freed‚ 1631864433528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636376760320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15749173 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636465420288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115534269020672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154304450989568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15749171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20041239 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155940933449216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155940827749888 % _timestamp‚1.7842393624886086e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15749173 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154304450989568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10546317 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152493339549680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154304468028928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15749171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚1304.352427397 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636376760320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1685560890368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154379600655872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20051059 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1686365749760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154379617695232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15756802 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818018 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20051059 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154200381829098 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1637162476032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15049906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156016868838400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4498533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15756800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10551377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10551373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1686499380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154200278003200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1637162476032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115590312638976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632734714666 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35807859 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152567647114432L¯jÞï&Ó éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154379617695232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15756802 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818018 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20051059 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154200381829098 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1637162476032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15049906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156016868838400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4498533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15756800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10551377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10551373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1686499380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154200278003200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1637162476032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115590312638976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632734714666 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35807859 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152567647114432 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156016868838400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15050085 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4498708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117276906129408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1637251143168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154379600655872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20051059 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1686365749760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200¢ÄOÿß2Ô - _wandbruntime‚1304  _runtime‚1304§e&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7611406445503235 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚4 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3809124231338501 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49424147605895996 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8878411054611206 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9837532043457031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17603126168251038 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6517006754875183 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9794127345085144 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.040264010429382324 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36342594027519226 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9588253498077393 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9946774840354919 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43667903542518616 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1675771027803421 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6828910708427429 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8074074983596802 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7056788802146912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7328402996063232 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5845561623573303 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9946774840354919 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015553143806755543 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_4‚0.7671635150909424 /particle/accuracy_3_of_4‚0.9874716401100159 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1338847428560257 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933041930198669 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07019499689340591 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7298844456672668 7"Purity/0FRt2SRqqt0FBt/event_purity‚0.38348388671875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37589797377586365 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16949494183063507 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3188668489456177 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.677121639251709 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12343333661556244 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5631939768791199 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924236536026001 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5455589294433594 +jet/accuracy_3_of_3‚0.05696186050772667 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8635938763618469 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07668434083461761 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9031985402107239 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19922105967998505 /particle/accuracy_4_of_4‚0.9312609434127808 trainer/global_step‚1999 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903432726860046 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43386396765708923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6646276712417603 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040958039462566376 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21054871380329132 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.43833333253860474 +particle/specificity‚0.9097476005554199 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9882939457893372 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22862684726715088 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07794784009456635 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6059679388999939 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9838202595710754 +jet/accuracy_3_of_4‚0.07323213666677475 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19743017852306366 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15113726258277893 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7332462668418884 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9989973902702332 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1752602756023407 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5058128833770752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7646105885505676 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3768477439880371 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8401910066604614 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6312735676765442 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021189333871006966 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19872397184371948 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8430523872375488 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9888193011283875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.5165860652923584 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 % _timestamp‚1.7842393765094543e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37166666984558105 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6577227115631104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19800935685634613 _step‚164 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8049131035804749 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6345693469047546 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6759259104728699 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3605705201625824 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8067271709442139 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9130463600158691 'particle/f_score‚0.6211516261100769 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902109503746033 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13803356885910034 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0320465974509716 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7367401123046875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07607660442590714 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1825079619884491 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18682177364826202 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912497401237488 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6339651346206665 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19139571487903595 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15914317965507507 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.49677449464797974 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029835939407348633 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6742375493049622 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9815192222595215 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5619998574256897 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07901164144277573 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6411193013191223 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26727738976478577 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2088991105556488 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17078173160552979 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5994874835014343 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *particle/sensitivity‚0.599202036857605 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18734094500541687 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6759259104728699 *jet/accuracy_1_of_2‚0.7853423357009888 particle/accuracy_2_of_3‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9882561564445496 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37254777550697327 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7977597117424011 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.984317421913147 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7377535700798035 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9928817749023438 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7094104290008545 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6528863310813904 *jet/accuracy_1_of_4‚0.9537683129310608 particle/accuracy_1_of_3‚1 *loss/val_total_loss‚11.290538850005012 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3017367124557495 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9780259132385254 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19872397184371948 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7977597117424011 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6143655776977539 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5253256559371948ھǮþ=Û ø{ @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _runtime‚1318.373821091 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26014378666877747 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960067272186279 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6283015608787537 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8306528925895691 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761523008346558 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4947916865348816 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20977090299129486 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7007556557655334 +jet/accuracy_2_of_2‚0.13269445300102234 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5495993494987488 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6347557902336121 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7611406445503235 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚4 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3809124231338501 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49424147605895996 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8878411054611206 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9837532043457031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17603126168251038 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6517006754875183 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9794127345085144 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.040264010429382324 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36342594027519226 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9588253498077393 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9946774840354919 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43667903542518616 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1675771027803421 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6828910708427429 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8074074983596802 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7056788802146912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7328402996063232 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5845561623573303 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9946774840354919 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015553143806755543 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_4‚0.7671635150909424 /particle/accuracy_3_of_4‚0.9874716401100159 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1338847428560257 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933041930198669 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07019499689340591 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7298844456672668 7"Purity/0FRt2SRqqt0FBt/event_purity‚0.38348388671875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37589797377586365 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16949494183063507 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3188668489456177 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.677121639251709 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12343333661556244 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5631939768791199 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924236536026001 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5455589294433594 +jet/accuracy_3_of_3‚0.05696186050772667 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8635938763618469 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07668434083461761 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9031985402107239 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19922105967998505 /particle/accuracy_4_of_4‚0.9312609434127808 trainer/global_step‚1999 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903432726860046 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43386396765708923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6646276712417603 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040958039462566376 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21054871380329132 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.43833333253860474 +particle/specificity‚0.9097476005554199 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9882939457893372 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22862684726715088 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07794784009456635 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6059679388999939 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9838202595710754 +jet/accuracy_3_of_4‚0.07323213666677475 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19743017852306366 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15113726258277893 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7332462668418884 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814815521240234 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9989973902702332 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1752602756023407 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5058128833770752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7646105885505676 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3768477439880371 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8401910066604614 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6312735676765442 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021189333871006966 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19872397184371948 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8430523872375488 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9888193011283875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.5165860652923584 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 % _timestamp‚1.7842393765094543e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37166666984558105 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6577227115631104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19800935685634613 _step‚164 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8049131035804749 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6345693469047546 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6759259104728699 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3605705201625824 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8067271709442139 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9130463600158691 'particle/f_score‚0.6211516261100769 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902109503746033 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13803356885910034 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0320465974509716 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7367401123046875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07607660442590714 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1825079619884491 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18682177364826202 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912497401237488 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6339651346206665 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19139571487903595 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15914317965507507 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.49677449464797974 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029835939407348633 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6742375493049622 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9815192222595215 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5619998574256897 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07901164144277573 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6411193013191223 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26727738976478577 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2088991105556488 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17078173160552979 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5994874835014343 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *particle/sensitivity‚0.599202036857605 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18734094500541687 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6759259104728699 *jet/accuracy_1_of_2‚0.7853423357009888 particle/accuracy_2_of_3‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9882561564445496 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37254777550697327 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7977597117424011 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.984317421913147 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7377535700798035 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9928817749023438 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7094104290008545 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6528863310813904 *jet/accuracy_1_of_4‚0.9537683129310608 particle/accuracy_1_of_3‚1 *loss/val_total_loss‚11.290538850005012 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3017367124557495 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9780259132385254 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19872397184371948 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7977597117424011 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6143655776977539 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5253256559371948 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7254740595817566 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814815521240234 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9944363832473755 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1485206037759781 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07019499689340591 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.679923415184021 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6979605555534363 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6704556941986084 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48200562596321106 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4004629850387573 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6411193013191223 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8061035871505737 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18307621777057648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.984364926815033 particle/accuracy_1_of_2‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20766861736774445 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 jet/accuracy_2_of_5‚1¤å*{^hÜ ‚(:Ý ’ªåÒ‘²ÞA disk.md0.in‚0 disk.dm-0.in‚ 121.3203125$ disk.nvme0c0n1.out‚ 26376.9140625" disk.nvme0n1.out‚ 26376.9140625 disk.nvme1c1n1.in‚ 2.73828125 disk.nvme1n1.in‚ 2.73828125$ disk.nvme1n1.out‚5882.1728515625* proc.memory.percent‚18.005037307739258" disk.nvme0c0n1.in‚ 118.58203125) disk./.usagePercent‚74.45381958425249% disk./.usageGB‚1143.0522651672363" disk.dm-0.out‚31586.8544921875 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚91.73882007598877) proc.memory.availableMB‚ 1353.51171875 network.sent‚ 2.895311e+06" proc.memory.rssMB‚ 2949.9453125 proc.cpu.threads‚16 disk.rbd0.out‚0 network.recv‚394492 disk.md0.out‚ 38.46484375 disk.dm-1.out‚ 589.90234375 disk.nvme0n1.in‚ 118.58203125& disk.nvme1c1n1.out‚5882.1728515625 cpu‚25.764454480969075Ϊü¸’Ý ‚(:‡ ’ªåÒø°›B gpu.0.gpu‚94.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚113.829' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.943 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>½¸½aÞ ‚(:Ö ¡ªåÒ®êA network.sent‚ 2.951801e+06& disk.nvme1c1n1.out‚6188.5673828125 disk.nvme1n1.in‚ 2.73828125 disk.md0.out‚ 39.0390625" disk.dm-0.out‚32455.1748046875 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚398774) disk./.usagePercent‚74.45080932159995% disk./.usageGB‚1143.0060501098633 disk.nvme1c1n1.in‚ 2.73828125$ disk.nvme1n1.out‚6188.5673828125 disk.dm-0.in‚ 121.3203125! disk.nvme0n1.out‚ 27003.984375$ memory_percent‚91.74096584320068 disk.dm-1.out‚653.8125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 118.58203125) proc.memory.availableMB‚ 1353.16015625" proc.memory.rssMB‚ 2949.9453125* proc.memory.percent‚18.005037307739258 proc.cpu.threads‚16 disk.md0.in‚0# disk.nvme0c0n1.out‚ 27003.984375 disk.nvme0n1.in‚ 118.58203125 cpu‚25.7626208615749458?kh’ß ‚(:‡ ¡ªåÒð‹ØA gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚191.283' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚63.761 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OË›7mà ‚(:â °ªåÒˆüÛA disk.dm-0.in‚ 121.3203125 disk.dm-1.out‚ 653.876953125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 118.58203125( disk.nvme0c0n1.out‚27007.58154296875 disk.nvme1n1.in‚ 2.73828125" disk.nvme1n1.out‚ 6189.32421875 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 6189.32421875# memory_percent‚91.7395830154419 cpu‚25.760835678618072 proc.cpu.threads‚16 network.sent‚ 2.956316e+06 disk.md0.in‚0 disk.md0.out‚ 39.37890625# disk.dm-0.out‚32458.72802734375 disk.nvme0n1.in‚ 118.58203125& disk.nvme0n1.out‚27007.58154296875 disk.nvme1c1n1.in‚ 2.73828125* proc.memory.percent‚18.005037307739258) disk./.usagePercent‚74.45080783075588% disk./.usageGB‚1143.0060272216797) proc.memory.availableMB‚ 1353.38671875" proc.memory.rssMB‚ 2949.9453125 network.recv‚401240ô€—á ‚(:’ °ªåÒ’¦ÉA ÙÙxE‚ gpu.0.gpu‚97.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.342' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.44733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊŸ¨2â - _wandbruntime‚1350  _runtime‚1350è”[ÛÈã  % _timestamp‚1.7842394082512221e+09 _runtime‚ 1350.11512893 _step‚165 ' lr-AdamW/pg1‚0.00023255794032943648 ' lr-AdamW/pg2‚0.00023255794032943648 trainer/global_step‚2049ËØ–‹Íä Ç trainer/global_step‚2049 % _timestamp‚1.7842394082512221e+09 _runtime‚ 1350.11512893 _step‚165 ' lr-AdamW/pg1‚0.00023255794032943648 ' lr-AdamW/pg2‚0.00023255794032943648¥1^›2å - _wandbruntime‚1350  _runtime‚1350c¥âÏß'æ ÙO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1691156077568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4650408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157181332855472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252421 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15557600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15557784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157181349894832 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120886150602240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10907376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10907372 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20626741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _runtime‚1350.115928321 _step‚166 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160749577678336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20616598 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159058421600768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36879164 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1686518222902 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691244737536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120886357378560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1741836752896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159058421600768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16252423 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1691156077568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119144520625664 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20626741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _runtime‚1350.115928321 _step‚166 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160749577678336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20616598 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159058421600768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36879164 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1686518222902 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691244737536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120886357378560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1741836752896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159058421600768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16252423 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1691156077568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119144520625664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120886150602240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175¦³ªŽå2è - _wandbruntime‚1350  _runtime‚1350럦@ê&é äM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159133571267072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260050 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20626418 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1692030460416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1742532115968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1687388504040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687301717502 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120943311517696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚167 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4652390 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1692030460416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120943096359936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10912428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20636575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160825618766848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10912432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15564818 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20626418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159133588306432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157255640420224 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158943028924264 % _timestamp‚1.7842394086487648e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260052 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1350.512915544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260050 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4652390 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1692030460416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120943096359936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_ðºd^!batch_end/inactive_split.large_pool.freed‚10912428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20636575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160825618766848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10912432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15564818 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20626418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159133588306432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157255640420224 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158943028924264 % _timestamp‚1.7842394086487648e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260052 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1350.512915544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260050 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159133571267072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260050 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20626418 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1692030460416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1742532115968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1687388504040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687301717502 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120943311517696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚167 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279§Çݯ2ë - _wandbruntime‚1350  _runtime‚1350~G-Íì  /loss/FBt1/detection_loss‚0.8405951469246565 /loss/FBt2/detection_loss‚1.1534005051851273 epoch‚5 1loss/FBt1/assignment_loss‚0.11717789211146237 % _timestamp‚1.7842394086497383e+09 _runtime‚1350.513468255 0loss/FRt1/assignment_loss‚2.6315049516351037 0loss/FRt2/assignment_loss‚1.3450868286641904 1loss/FBt2/assignment_loss‚0.12913878588607075 trainer/global_step‚2049 2loss/SRqqt2/assignment_loss‚0.7306882636860552 1loss/SRqqt2/detection_loss‚0.7328114305546287 &loss/total_loss‚11.583427601657922 _step‚168 2loss/SRqqt1/assignment_loss‚0.6528698649734581 /loss/FRt1/detection_loss‚0.5781976306359702 .loss/FRt2/detection_loss‚1.459203775687368 1loss/SRqqt1/detection_loss‚1.2127525257138314ND í † 0loss/FRt1/assignment_loss‚2.6315049516351037 0loss/FRt2/assignment_loss‚1.3450868286641904 1loss/FBt2/assignment_loss‚0.12913878588607075 trainer/global_step‚2049 2loss/SRqqt2/assignment_loss‚0.7306882636860552 1loss/SRqqt2/detection_loss‚0.7328114305546287 &loss/total_loss‚11.583427601657922 _step‚168 2loss/SRqqt1/assignment_loss‚0.6528698649734581 /loss/FRt1/detection_loss‚0.5781976306359702 .loss/FRt2/detection_loss‚1.459203775687368 1loss/SRqqt1/detection_loss‚1.2127525257138314 /loss/FBt1/detection_loss‚0.8405951469246565 /loss/FBt2/detection_loss‚1.1534005051851273 epoch‚5 1loss/FBt1/assignment_loss‚0.11717789211146237 % _timestamp‚1.7842394086497383e+09 _runtime‚1350.513468255¨™F§ünî ‚(:ã ¿ªåÒÇ­’B) disk./.usagePercent‚74.45080633991181 disk.nvme1c1n1.in‚ 2.73828125 disk.md0.in‚0 disk.dm-0.in‚ 121.3203125" proc.memory.rssMB‚ 2949.9453125 network.sent‚ 2.961279e+06 network.recv‚404865$ disk./.usageGB‚1143.006004333496( disk.nvme0c0n1.out‚27010.53564453125 disk.nvme0n1.in‚ 118.58203125& disk.nvme0n1.out‚27010.53564453125# disk.dm-0.out‚32460.89599609375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 118.58203125& disk.nvme1c1n1.out‚6190.5712890625$ disk.nvme1n1.out‚6190.5712890625 disk.md0.out‚ 40.296875 disk.dm-1.in‚0 disk.dm-1.out‚ 653.994140625 disk.rbd0.in‚0( proc.memory.availableMB‚ 1353.5078125 disk.nvme1n1.in‚ 2.73828125$ memory_percent‚91.73884391784668* proc.memory.percent‚18.005037307739258 cpu‚25.759431119779347 proc.cpu.threads‚16F¥Ûï ‚(:’ ¿ªåÒݾüB gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.634' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.54466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pߟið ‚(:Þ ΪåÒæêóA disk.dm-0.in‚ 121.3203125* proc.memory.percent‚18.005037307739258 cpu‚25.75790306537219) disk./.usagePercent‚74.45132763838849# disk.nvme1n1.out‚6191.634765625 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 654.087890625' proc.memory.availableMB‚ 1353.421875 proc.cpu.threads‚16 disk.rbd0.in‚0 disk.nvme0n1.in‚ 118.58203125 disk.nvme1c1n1.in‚ 2.73828125% disk.nvme1c1n1.out‚6191.634765625# memory_percent‚91.7393684387207" proc.memory.rssMB‚ 2949.9453125 network.sent‚ 3.021924e+06% disk./.usageGB‚1143.0140075683594' disk.nvme0c0n1.out‚27014.6455078125% disk.nvme0n1.out‚27014.6455078125 disk.nvme1n1.in‚ 2.73828125" disk.dm-0.out‚32465.5361328125 network.recv‚408657 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 118.58203125 disk.md0.out‚ 40.48828125¼+h’ñ ‚(:‡ ΪåÒ­‹¬B gpu.0.gpu‚56.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚227.571' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.857 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¸2ò - _wandbruntime‚1381  _runtime‚1381”Ì¥.Èó  ' lr-AdamW/pg1‚0.00023230652223705684 ' lr-AdamW/pg2‚0.00023230652223705684 trainer/global_step‚2099 % _timestamp‚1.7842394394508197e+09 _runtime‚ 1381.31480093 _step‚169»¯… Íô Ç ' lr-AdamW/pg1‚0.00023230652223705684 ' lr-AdamW/pg2‚0.00023230652223705684 trainer/global_step‚2099 % _timestamp‚1.7842394394508197e+09 _runtime‚ 1381.31480093 _step‚169©‚yb 2õ - _wandbruntime‚1381  _runtime‚1381µyùà'ö ÚO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123733832140288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162815904915968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162815921955328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21119593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1730506017792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725843540136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1730506017792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15918989 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1730594677760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21109450 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16633871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162815904915968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162621719704568 _runtime‚1381.318335125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16633873 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164546410933760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121951407025664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164546516633088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160895876164432 % _timestamp‚1.7842394394517176e+09 _step‚170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164546516633088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15919172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37743321 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123733612781056 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162815921955328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1730594677760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16633873 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121951312915968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160895859125072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16633871 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11160172 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21109450 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164546410933760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4758817 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1782299865088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162621615878782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21119593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11160176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4758996 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37753466 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048°r.’ë#÷ ßO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21109450 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164546410933760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4758817 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1782299865088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162621615878782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21119593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11160176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4758996 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37753466 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123733832140288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162815904915968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162815921955328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21119593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1730506017792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725843540136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1730506017792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15918989 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1730594677760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21109450 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16633871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162815904915968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162621719704568 _runtime‚1381.318335125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16633873 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164546410933760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121951407025664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164546516633088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160895876164432 % _timestamp‚1.7842394394517176e+09 _step‚170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164546516633088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15919172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37743321 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123733612781056 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162815921955328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1730594677760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16633873 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121951312915968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160895859125072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16633871 KEDeviceStatsMonitor.on_train_']¤[úbatch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11160172 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760ª.r2ø - _wandbruntime‚1381  _runtime‚1381BceVê&ù äM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1783238519296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37770929 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164622346315776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16641500 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160970166689824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15926386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21119270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162891054582272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731291733504 % _timestamp‚1.7842394398478646e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162696793724672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15926204 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160970149650464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16641500 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16641502 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1731380400640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11165228 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123790783272960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1726627034848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1726540248310 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1731380400640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2099 _runtime‚1381.711985986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11165232 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21129427 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162891071621632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4761154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚171 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162696689898774 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122007450643968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1731291733504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21119270 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164622452022272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164622452022272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162891054582272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16641502 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21129427 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122007544753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162891071621632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123790559726592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1783109082624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2099 _runtime‚1381.711985986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11165232 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21129427 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162891071621632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4761154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚171 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162696689898774 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122007450643968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1731291733504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21119270 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164622452022272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164622452022272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162891054582272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16641502 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21129427 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122007544753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162891071621632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123790559726592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1783109082624 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21119270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162891054582272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731291733504 % _timestamp‚1.7842394398478646e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162696793724672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15926204 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160970149650464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16641500 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0«tú·82û - _wandbruntime‚1381  _runtime‚1381ý.Lü þ /loss/FRt1/assignment_loss‚2.507123935132705 0loss/FRt2/assignment_loss‚1.3672491916649634 0loss/FBt2/assignment_loss‚0.1658551879008964 .loss/FRt2/detection_loss‚1.337280252614442 1loss/FBt1/assignment_loss‚0.07440750643315407 1loss/SRqqt2/detection_loss‚0.7560346362903944 /loss/FBt2/detection_loss‚1.8907014434891087 1loss/SRqqt2/assignment_loss‚0.742118109005694 /loss/FBt1/detection_loss‚1.0413909177248706 &loss/total_loss‚12.415699472065368 epoch‚5 _step‚172 2loss/SRqqt1/assignment_loss‚0.7357984620056606 /loss/FRt1/detection_loss‚0.5810479811733127 1loss/SRqqt1/detection_loss‚1.2166918486301668 trainer/global_step‚2099 % _timestamp‚1.7842394398488371e+09 _runtime‚1381.7124634970$ ý ƒ 1loss/FBt1/assignment_loss‚0.07440750643315407 1loss/SRqqt2/detection_loss‚0.7560346362903944 /loss/FBt2/detection_loss‚1.8907014434891087 1loss/SRqqt2/assignment_loss‚0.742118109005694 /loss/FBt1/detection_loss‚1.0413909177248706 &loss/total_loss‚12.415699472065368 epoch‚5 _step‚172 2loss/SRqqt1/assignment_loss‚0.7357984620056606 /loss/FRt1/detection_loss‚0.5810479811733127 1loss/SRqqt1/detection_loss‚1.2166918486301668 trainer/global_step‚2099 % _timestamp‚1.7842394398488371e+09 _runtime‚1381.712463497 /loss/FRt1/assignment_loss‚2.507123935132705 0loss/FRt2/assignment_loss‚1.3672491916649634 0loss/FBt2/assignment_loss‚0.1658551879008964 .loss/FRt2/detection_loss‚1.337280252614442¬Å~&Xlþ ‚(:á ݪåÒš£ÝA disk.dm-0.in‚ 121.3203125 disk.dm-1.in‚0$ memory_percent‚91.73521995544434* proc.memory.percent‚18.005037307739258% disk./.usageGB‚1143.0141792297363 disk.nvme1c1n1.in‚ 2.73828125 disk.md0.out‚ 41.26953125 disk.dm-1.out‚ 654.166015625 disk.rbd0.in‚0 cpu‚25.756582842483855 proc.cpu.threads‚16 network.sent‚ 3.027838e+06 network.recv‚412141" disk.nvme0c0n1.in‚ 118.58203125& disk.nvme0c0n1.out‚27017.673828125$ disk.nvme0n1.out‚27017.673828125 disk.nvme1n1.in‚ 2.73828125$ disk.nvme1n1.out‚6192.6865234375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 118.58203125& disk.nvme1c1n1.out‚6192.6865234375" disk.dm-0.out‚32467.9189453125( proc.memory.availableMB‚ 1354.1015625" proc.memory.rssMB‚ 2949.9453125) disk./.usagePercent‚74.45133881971903 disk.md0.in‚0.4" šÿ ‚(: ݪåÒ…ð¢B gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚194.62' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.87333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ³z'r€ ‚(:ç ìªåÒà¡ÙA disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚27021.1142578125 disk.nvme0n1.in‚ 118.58203125" proc.memory.rssMB‚ 2949.9453125 network.recv‚416129) disk./.usagePercent‚74.45138826604737% disk.nvme1n1.out‚6193.57763671875 disk.md0.out‚ 41.7109375 disk.dm-0.in‚ 121.3203125" disk.nvme0c0n1.in‚ 118.58203125% disk.nvme0n1.out‚27021.1142578125 proc.cpu.threads‚16 disk.nvme1n1.in‚ 2.73828125! disk.dm-1.out‚654.25146484375) proc.memory.availableMB‚ 1353.78515625* proc.memory.percent‚18.005037307739258" disk.dm-0.out‚32471.2177734375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚ 2.73828125' disk.nvme1c1n1.out‚6193.57763671875$ memory_percent‚91.73715114593506 cpu‚25.754933539792418 network.sent‚ 3.090192e+06% disk./.usageGB‚1143.0149383544922 disk.md0.in‚0Ù}’ ‚(:‡ ìªåÒǪ‘B gpu.0.gpu‚89.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚133.071' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.357 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø‘Û2‚ - _wandbruntime‚1412  _runtime‚1412Ú>ǃ Á _step‚173 & lr-AdamW/pg1‚0.0002320477838810683 & lr-AdamW/pg2‚0.0002320477838810683 trainer/global_step‚2149 % _timestamp‚1.7842394710946968e+09 _runtime‚1412.958664508rÓ¥cÌ„ Æ % _timestamp‚1.7842394710946968e+09 _runtime‚1412.958664508 _step‚173 & lr-AdamW/pg1‚0.0002320477838810683 & lr-AdamW/pg2‚0.0002320477838810683 trainer/global_step‚2149­P± 2… - _wandbruntime‚1412  _runtime‚1412a˜µôK† ÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38617623 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1823014361600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21612445 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1764995284518 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015321 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4866296 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1769855958016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11412972 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166375380679190 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11412976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21602302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015323 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166573405270528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1769944617984 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168343349888512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚174 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168343349888512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16279096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4866124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124758293425ؤœÔ• 664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164610402434032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124758199315968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164610385394672 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 % _timestamp‚1.7842394710955753e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166573388231168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126581090525184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166375484504976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1822891209216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168343244189184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21612445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1765082070944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21602302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1769944617984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769855958016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130`9?På'‡ ßO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166573388231168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166573405270528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168343244189184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015321 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16279272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38627768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015323 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166573405270528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1769944617984 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168343349888512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚174 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168343349888512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16279096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4866124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124758293425664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164610402434032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124758199315968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164610385394672 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 % _timestamp‚1.7842394710955753e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166573388231168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126581090525184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166375484504976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1822891209216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168343244189184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21612445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1765082070944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21602302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1769944617984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769855958016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38617623 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1823014361600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21612445 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1764995284518 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015321 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4866296 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1769855958016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11412972 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166375380679190 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11412976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21602302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126638249736704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168419179571200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124814337043968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1770730340864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166648554936832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1823818583040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11418032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17022950 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7842394714919739e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166648537897472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 trainer/global_step‚2149 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168419285277696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17022952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1765778779118 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17022950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4868244 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1823705923584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164684692959424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1770730340864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166648554936832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1765865565656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚1413.356118451 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4868425)ÛɦG$Š éM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17022950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4868244 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1823705923584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164684692959424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1770730340864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166648554936832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1765865565656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚1413.356118451 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4868425 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126638249736704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168419179571200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124814337043968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1770730340864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166648554936832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1823818583040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11418032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17022950 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7842394714919739e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16664853÷£Y¨7897472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 trainer/global_step‚2149 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168419285277696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17022952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1765778779118¯%Ø,2‹ - _wandbruntime‚1413  _runtime‚1413œToŒ û 0loss/FRt1/assignment_loss‚2.5990267801615925 2loss/SRqqt1/assignment_loss‚0.7168331217820473 1loss/FBt2/assignment_loss‚0.41221691660295645 &loss/total_loss‚13.920848384245181 /loss/FRt2/assignment_loss‚1.369307427996889 1loss/SRqqt1/detection_loss‚1.3003002003402166 trainer/global_step‚2149 1loss/FBt1/assignment_loss‚0.11953550069850723 1loss/SRqqt2/detection_loss‚0.7143228633069227 /loss/FBt2/detection_loss‚1.7829602869420216 epoch‚5 _step‚176 0loss/SRqqt2/assignment_loss‚0.69324797596656 /loss/FRt1/detection_loss‚0.5929879810213842 ,loss/FRt2/detection_loss‚1.4578587913066 /loss/FBt1/detection_loss‚2.1622505381194843 $ _timestamp‚1.784239471492971e+09 _runtime‚1413.356633871kAî£ € _runtime‚1413.356633871 0loss/FRt1/assignment_loss‚2.5990267801615925 2loss/SRqqt1/assignment_loss‚0.7168331217820473 1loss/FBt2/assignment_loss‚0.41221691660295645 &loss/total_loss‚13.920848384245181 /loss/FRt2/assignment_loss‚1.369307427996889 1loss/SRqqt1/detection_loss‚1.3003002003402166 trainer/global_step‚2149 1loss/FBt1/assignment_loss‚0.11953550069850723 1loss/SRqqt2/detection_loss‚0.7143228633069227 /loss/FBt2/detection_loss‚1.7829602869420216 epoch‚5 _step‚176 0loss/SRqqt2/assignment_loss‚0.69324797596656 /loss/FRt1/detection_loss‚0.5929879810213842 ,loss/FRt2/detection_loss‚1.4578587913066 /loss/FBt1/detection_loss‚2.1622505381194843 $ _timestamp‚1.784239471492971e+09°Iã‡b’Ž ‚(:‡ ûªåÒ¬õîA gpu.0.gpu‚92.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚193.515' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚64.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)‡îTm ‚(:â ûªåÒùªþA disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚27023.748046875& proc.memory.availableMB‚ 1353.65625* proc.memory.percent‚18.005037307739258 cpu‚25.753654211898766" disk.dm-0.out‚32473.7021484375 disk.nvme0n1.in‚ 118.58203125 disk.nvme1c1n1.in‚ 2.73828125 disk.nvme1n1.in‚ 2.73828125 disk.md0.out‚ 42.20703125" disk.nvme0c0n1.in‚ 118.58203125 network.sent‚ 3.095018e+06% disk./.usageGB‚1143.0152893066406$ disk.nvme0n1.out‚27023.748046875# memory_percent‚91.7379379272461" proc.memory.rssMB‚ 2949.9453125 proc.cpu.threads‚16 disk.dm-0.in‚ 121.3203125 network.recv‚419731' disk.nvme1c1n1.out‚6194.60986328125% disk.nvme1n1.out‚6194.60986328125 disk.md0.in‚0! disk.dm-1.out‚654.36865234375 disk.rbd0.out‚0) disk./.usagePercent‚74.45141112565645KŒYür ‚(:ç Š«åÒÇ¢øA disk.dm-0.in‚ 121.3203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 118.58203125 disk.nvme1c1n1.in‚ 2.73828125' disk.nvme1c1n1.out‚6195.42822265625 disk.nvme1n1.in‚ 2.73828125# disk.dm-0.out‚32477.08056640625" disk.nvme0c0n1.in‚ 118.58203125% disk.nvme1n1.out‚6195.42822265625$ memory_percent‚91.73755645751953& proc.memory.availableMB‚ 1353.71875" proc.memory.rssMB‚ 2949.9453125 network.sent‚ 3.157915e+06! disk.dm-1.out‚654.47119140625 cpu‚25.752069819952027 proc.cpu.threads‚16 network.recv‚423798) disk./.usagePercent‚74.45142404630506 disk.md0.in‚0 disk.md0.out‚ 42.5234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27027.09228515625& disk.nvme0n1.out‚27027.09228515625* proc.memory.percent‚18.005037307739258% disk./.usageGB‚1143.0154876708984¤Üž‘ ‚(:“ Š«åÒ·©²B gpu.0.gpu‚82.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.275' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.425000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p2’ - _wandbruntime‚1444  _runtime‚1444guúÐÅ“ ¿ _runtime‚1444.582396841 _step‚177 % lr-AdamW/pg1‚0.000231781741879752 % lr-AdamW/pg2‚0.000231781741879752 trainer/global_step‚2199 % _timestamp‚1.7842395027168741e+09˽ ãÊ” Ä % lr-AdamW/pg1‚0.000231781741879752 % lr-AdamW/pg2‚0.000231781741879752 trainer/global_step‚2199 % _timestamp‚1.7842395027168741e+09 _runtime‚1444.582396841 _step‚177±þÇ2• - _wandbruntime‚1444  _runtime‚1444„‹¨¾à'– ÚO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168324911664272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22105297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172140183143936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170330871546368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22095154 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1809294558208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17396771 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170330888585728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129428662999040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1863360020992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 % _timestamp‚1.7842395027174997e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17396773 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168324928703632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170330871546368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172140183143936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127565085715968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1804320601752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172140077444608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22095154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804233815326 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172140077444608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170330888585728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1809205898240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11665772 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1804320601752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172140077444608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22095154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804233815326 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172140077444608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170330888585728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1809205898240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11665772 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168324911664272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22105297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172140183143936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170330871546368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22095154 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1809294558208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17396771 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170330888585728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129428662999040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1863360020992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 % _timestamp‚1.7842395027174997e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17396773 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168324928703632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170330871546368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172140183143936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127565085715968 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842395031125884e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16646206 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1864297511936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129485389422080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚179 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170204219499590 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1864165978112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1805104096464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170406038252032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4975378 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127621317553664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17404402 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170406021212672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22104974 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22104974 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168399202189664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17404400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11670832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168399219229024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_ÅRóý4 split.small_pool.allocated‚4975561 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170204323325488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172216118533120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22115131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810080281088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚1444.976715274 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172216012826624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1809991613952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11670828 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538&&e‹ï&š éM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1864165978112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1805104096464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170406038252032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4975378 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127621317553664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17404402 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170406021212672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22104974 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22104974 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168399202189664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17404400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11670832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168399219229024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4975561 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170204323325488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172216118533120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22115131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810080281088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚1444.976715274 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172216012826624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1809991613952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11670828 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16646393 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842395031125884e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16646206 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1864297511936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129485389422080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚179 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170204219499590 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812³DÏ;2› - _wandbruntime‚1444  _runtime‚1444¬*.œ ÿ /loss/FBt2/detection_loss‚1.3925007904569309 epoch‚5 0loss/FRt2/assignment_loss‚1.2974042745543197 /loss/FRt1/detection_loss‚0.5744158645656205 1loss/SRqqt1/detection_loss‚1.3183405876208885 2loss/SRqqt1/assignment_loss‚0.6843702298273404 1loss/FBt1/assignment_loss‚0.11562350957586286 0loss/FBt2/assignment_loss‚0.4216224517578616 /loss/FBt1/detection_loss‚1.1603747328752414 trainer/global_step‚2199 $ _timestamp‚1.784239503113553e+09 _runtime‚1444.977267434 0loss/FRt1/assignment_loss‚2.4828358519263984 2loss/SRqqt2/assignment_loss‚0.7156494880560044 /loss/FRt2/detection_loss‚1.4124561563785136 0loss/SRqqt2/detection_loss‚0.771935654759324 &loss/total_loss‚12.347529592354306 _step‚180¯O>  „ 0loss/FRt2/assignment_loss‚1.2974042745543197 /loss/FRt1/detection_loss‚0.5744158645656205 1loss/SRqqt1/detection_loss‚1.3183405876208885 2loss/SRqqt1/assignment_loss‚0.6843702298273404 1loss/FBt1/assignment_loss‚0.11562350957586286 0loss/FBt2/assignment_loss‚0.4216224517578616 /loss/FBt1/detection_loss‚1.1603747328752414 trainer/global_step‚2199 $ _timestamp‚1.784239503113553e+09 _runtime‚1444.977267434 0loss/FRt1/assignment_loss‚2.4828358519263984 2loss/SRqqt2/assignment_loss‚0.7156494880560044 /loss/FRt2/detection_loss‚1.4124561563785136 0loss/SRqqt2/detection_loss‚0.771935654759324 &loss/total_loss‚12.347529592354306 _step‚180 /loss/FBt2/detection_loss‚1.3925007904569309 epoch‚5´µ’‚¿qž ‚(:æ ™«åÒ¿§B* proc.memory.percent‚18.005037307739258 network.sent‚ 3.162245e+06 disk.nvme1n1.in‚ 2.73828125 disk.md0.in‚0 disk.md0.out‚ 43.1796875 disk.dm-1.in‚0# memory_percent‚91.7367935180664 network.recv‚427261) disk./.usagePercent‚74.45164966070774 disk.nvme1c1n1.in‚ 2.73828125 disk.dm-0.in‚ 121.3203125# disk.dm-0.out‚32479.30712890625! disk.dm-1.out‚654.56103515625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27029.75439453125' disk.nvme1c1n1.out‚6196.45361328125% disk.nvme1n1.out‚6196.45361328125" disk.nvme0c0n1.in‚ 118.58203125 disk.nvme0n1.in‚ 118.58203125& proc.memory.availableMB‚ 1353.84375" proc.memory.rssMB‚ 2949.9453125 cpu‚25.750679430524222 proc.cpu.threads‚16% disk./.usageGB‚1143.0189514160156 disk.rbd0.out‚0& disk.nvme0n1.out‚27029.75439453125µwœŸ ‚(:‘ ™«åÒôÔùB gpu.0.gpu‚52.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚195.962' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.32066666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÔÙp  ‚(:å ¨«åÒ‹®B network.recv‚431506 disk.nvme0n1.in‚ 118.58203125 disk.md0.in‚0 disk.md0.out‚ 43.56640625" disk.dm-0.out‚32482.1083984375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚91.73719882965088 network.sent‚ 3.22467e+06 disk.dm-0.in‚ 121.3203125 disk.dm-1.in‚0* proc.memory.percent‚18.005037307739258) disk./.usagePercent‚74.45165363629194$ disk.nvme0n1.out‚27032.630859375 disk.nvme1c1n1.in‚ 2.73828125 disk.nvme1n1.in‚ 2.73828125& disk.nvme0c0n1.out‚27032.630859375) proc.memory.availableMB‚ 1353.77734375 cpu‚25.749496811271356% disk./.usageGB‚1143.0190124511719' disk.nvme1c1n1.out‚6197.26513671875% disk.nvme1n1.out‚6197.26513671875! disk.dm-1.out‚654.62548828125" disk.nvme0c0n1.in‚ 118.58203125" proc.memory.rssMB‚ 2949.9453125 proc.cpu.threads‚16²Èv¬›¡ ‚(: ¨«åÒÿ‹¹B gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚220.279' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.42633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03¾’%2¢ - _wandbruntime‚1475  _runtime‚14754ŽýÈ£  ' lr-AdamW/pg1‚0.00023150841332048852 ' lr-AdamW/pg2‚0.00023150841332048852 trainer/global_step‚2249 $ _timestamp‚1.784239534063017e+09 _runtime‚1475.928393332 _step‚181›ÙðÕͤ Ç ' lr-AdamW/pg1‚0.00023150841332048852 ' lr-AdamW/pg2‚0.00023150841332048852 trainer/global_step‚2249 $ _timestamp‚1.784239534063017e+09 _runtime‚1475.928393332 _step‚181µÈø?2¥ - _wandbruntime‚1475  _runtime‚1475:Øpãà'¦ ÚO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130371972115968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22598149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175937016399360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174088354861568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1848555838464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17778221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130372066225664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175937016399360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17778221 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17778223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843559132560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132276055312384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132275833856000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173882910280006 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _runtime‚1475.928456882 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175936910700032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1848644498432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚182 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5080351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174088371900928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚2249 % _timestamp‚1.7842395340636165e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174088354861568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11918572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1848644498432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175936910700032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5080172 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172039437933872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172039454973232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22588006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17778223 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174088371900928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11918576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1903861740032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22598149 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16998744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173883014105792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843472346134 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚101437¢sB§ ßO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173883014105792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843472346134 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130371972115968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22598149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175937016399360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174088354861568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1848555838464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17778221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130372066225664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175937016399360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17778221 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17778223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843559132560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132276055312384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132275833856000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173882910280006 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _runtime‚1475.928456882 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175936910700032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1848644498432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚182 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 120650e’=Û£26048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5080351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174088371900928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚2249 % _timestamp‚1.7842395340636165e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174088354861568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11918572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1848644498432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175936910700032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5080172 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172039437933872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172039454973232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22588006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17778223 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174088371900928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11918576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1903861740032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22598149 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16998744¶:—w2¨ - _wandbruntime‚1476  _runtime‚1476xwCé&© ãM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17006017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174163504527872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1904664920064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17005839 _runtime‚1476.323849535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130428203953664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174163521567232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132333002504704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132332774764032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22597826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11923628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11923632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1904798551040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785850 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22597826 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176012846082048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844342627272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785852 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785852 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174163504527872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22607983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚2249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5082385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172113745498624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22607983 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0<ƒå‘î&ª èM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5082211 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11923632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1904798551040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785850 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22597826 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176012846082048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844342627272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785852 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785852 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174163504527872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22607983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚2249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5082385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172113745498624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22607983 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17006017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174163504527872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1904664920064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17005839 _runtime‚1476.323849535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130428203953664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174163521567232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132333002504704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132332774764032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22597826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11923628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177845855216128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1944487839744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18159671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133178952625664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175753981242832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135123440465408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18159673 [IDeviceStatsMonitor.on_train_batch_start/active_bytesâKjP.small_pool.allocated‚ 1887994438656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1887994438656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5186430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179733743955456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚163 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133178858515968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23091001 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177845838176768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5186593 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1887905778688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179733743955456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23080858 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12171372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175753964203472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177845838176768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41240529 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179733849654784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17357802 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5186430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179733743955456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚163 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133178858515968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23091001 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177845838176768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5186593 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1887905778688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179733743955456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23080858 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12171372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175753964203472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177845838176768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41240529 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179733849654784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17357802 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177845855216128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1944487839744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18159671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133178952625664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175753981242832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135123440465408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488ºO3§!2¸ - _wandbruntime‚1508  _runtime‚1508Q·ìê&¹ äM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12176432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177920987843072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5188631 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23100835 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17365059 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17365235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12176428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚1508.013511878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18167300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23090678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133234996243968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18167300 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1888691494400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135180383963136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23100835 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175828254728864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1883494371542 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133235090353664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1883581158080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23090678 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1888780161536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179809679337472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1888780161536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135180162513920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚172 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177921004882432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17365059 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17365235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12176428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 A9DeviceStatsMonitor.on